Nova Resource:Tools/SAL

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.45.0-wmf.8</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Nova Resource:Tools/SAL</title>
    <ns>498</ns>
    <id>3086</id>
    <revision>
      <id>2320864</id>
      <parentid>2320849</parentid>
      <timestamp>2025-07-06T16:28:24Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8</comment>
      <origin>2320864</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="564898" sha1="qj83zr0ikuc1as4xf7p88wnnqf91hys" xml:space="preserve">=== 2025-07-06 ===
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>qj83zr0ikuc1as4xf7p88wnnqf91hys</sha1>
    </revision>
    <revision>
      <id>2320865</id>
      <parentid>2320864</parentid>
      <timestamp>2025-07-06T16:38:22Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8</comment>
      <origin>2320865</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="565043" sha1="d0nkthhs6bgyasg8npj3ymzq1v2tryh" xml:space="preserve">=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>d0nkthhs6bgyasg8npj3ymzq1v2tryh</sha1>
    </revision>
    <revision>
      <id>2320895</id>
      <parentid>2320865</parentid>
      <timestamp>2025-07-07T08:21:19Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging</comment>
      <origin>2320895</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="565164" sha1="6cntedu106pxnnhsoikfw18p7ag2kmy" xml:space="preserve">=== 2025-07-07 ===
* 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>6cntedu106pxnnhsoikfw18p7ag2kmy</sha1>
    </revision>
    <revision>
      <id>2320896</id>
      <parentid>2320895</parentid>
      <timestamp>2025-07-07T08:26:34Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging</comment>
      <origin>2320896</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="565285" sha1="jqk5xcqj4kp7rdwb3fr36j4fnooi1ad" xml:space="preserve">=== 2025-07-07 ===
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>jqk5xcqj4kp7rdwb3fr36j4fnooi1ad</sha1>
    </revision>
    <revision>
      <id>2320926</id>
      <parentid>2320896</parentid>
      <timestamp>2025-07-07T11:23:36Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld</comment>
      <origin>2320926</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="565393" sha1="hfp3qs8r95kxv7v4l2fuswtwk3h61sa" xml:space="preserve">=== 2025-07-07 ===
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>hfp3qs8r95kxv7v4l2fuswtwk3h61sa</sha1>
    </revision>
    <revision>
      <id>2320927</id>
      <parentid>2320926</parentid>
      <timestamp>2025-07-07T11:23:56Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld</comment>
      <origin>2320927</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="565520" sha1="h6tq20wzequtwll419e591npzmkolsj" xml:space="preserve">=== 2025-07-07 ===
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>h6tq20wzequtwll419e591npzmkolsj</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Toolsbeta/SAL</title>
    <ns>498</ns>
    <id>6726</id>
    <revision>
      <id>2320893</id>
      <parentid>2320631</parentid>
      <timestamp>2025-07-07T08:19:17Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging</comment>
      <origin>2320893</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="481441" sha1="2tfuxxmxvcjza7l8jbesszjwvzva9sj" xml:space="preserve">=== 2025-07-07 ===
* 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-03 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-02 ===
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-06-26 ===
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-24 ===
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging
* 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-06-23 ===
* 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-19 ===
* 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-18 ===
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-06-17 ===
* 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-16 ===
* 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-12 ===
* 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-11 ===
* 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-10 ===
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]])
* 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]])

=== 2025-06-09 ===
* 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>2tfuxxmxvcjza7l8jbesszjwvzva9sj</sha1>
    </revision>
    <revision>
      <id>2320894</id>
      <parentid>2320893</parentid>
      <timestamp>2025-07-07T08:19:33Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging</comment>
      <origin>2320894</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="481561" sha1="hv65gag538b4ppkaz9emuczr9w8h5wq" xml:space="preserve">=== 2025-07-07 ===
* 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-03 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-02 ===
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-06-26 ===
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-24 ===
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging
* 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-06-23 ===
* 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-19 ===
* 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-18 ===
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-06-17 ===
* 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-16 ===
* 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-12 ===
* 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-11 ===
* 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-10 ===
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]])
* 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]])

=== 2025-06-09 ===
* 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>hv65gag538b4ppkaz9emuczr9w8h5wq</sha1>
    </revision>
    <revision>
      <id>2320922</id>
      <parentid>2320894</parentid>
      <timestamp>2025-07-07T11:11:34Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld</comment>
      <origin>2320922</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="481669" sha1="q7usuzc0g30cmbamw6zu7f8laknucqd" xml:space="preserve">=== 2025-07-07 ===
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-03 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-02 ===
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-06-26 ===
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-24 ===
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging
* 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-06-23 ===
* 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-19 ===
* 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-18 ===
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-06-17 ===
* 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-16 ===
* 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-12 ===
* 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-11 ===
* 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-10 ===
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]])
* 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]])

=== 2025-06-09 ===
* 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>q7usuzc0g30cmbamw6zu7f8laknucqd</sha1>
    </revision>
    <revision>
      <id>2320923</id>
      <parentid>2320922</parentid>
      <timestamp>2025-07-07T11:22:50Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld</comment>
      <origin>2320923</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="481796" sha1="1qtwjjr4whedj8fpqq389ioy0gckqj1" xml:space="preserve">=== 2025-07-07 ===
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-03 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-02 ===
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-06-26 ===
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-24 ===
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging
* 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-06-23 ===
* 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-19 ===
* 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-18 ===
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-06-17 ===
* 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-16 ===
* 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-12 ===
* 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-11 ===
* 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-10 ===
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]])
* 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]])

=== 2025-06-09 ===
* 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>1qtwjjr4whedj8fpqq389ioy0gckqj1</sha1>
    </revision>
    <revision>
      <id>2320924</id>
      <parentid>2320923</parentid>
      <timestamp>2025-07-07T11:23:01Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld</comment>
      <origin>2320924</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="481904" sha1="145i0mbnrpbki5sd8kfom532rventgj" xml:space="preserve">=== 2025-07-07 ===
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-03 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-02 ===
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-06-26 ===
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-24 ===
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging
* 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-06-23 ===
* 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-19 ===
* 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-18 ===
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-06-17 ===
* 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-16 ===
* 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-12 ===
* 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-11 ===
* 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-10 ===
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]])
* 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]])

=== 2025-06-09 ===
* 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>145i0mbnrpbki5sd8kfom532rventgj</sha1>
    </revision>
    <revision>
      <id>2320925</id>
      <parentid>2320924</parentid>
      <timestamp>2025-07-07T11:23:18Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld</comment>
      <origin>2320925</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="482031" sha1="10lm7ec5wta8ah8tf6fg3sr4de1k5ax" xml:space="preserve">=== 2025-07-07 ===
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-03 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-02 ===
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-06-26 ===
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-24 ===
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging
* 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-06-23 ===
* 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-19 ===
* 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-18 ===
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-06-17 ===
* 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-06-16 ===
* 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-12 ===
* 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-11 ===
* 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-10 ===
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]])
* 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]])

=== 2025-06-09 ===
* 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>10lm7ec5wta8ah8tf6fg3sr4de1k5ax</sha1>
    </revision>
  </page>
  <page>
    <title>Server Admin Log</title>
    <ns>0</ns>
    <id>7919</id>
    <revision>
      <id>2320868</id>
      <parentid>2320842</parentid>
      <timestamp>2025-07-07T06:29:05Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply</comment>
      <origin>2320868</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="142844" sha1="e42b5p8q7ug6mrnvpf0f6bg2ir4j5jm" xml:space="preserve">== 2025-07-07 ==
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>e42b5p8q7ug6mrnvpf0f6bg2ir4j5jm</sha1>
    </revision>
    <revision>
      <id>2320870</id>
      <parentid>2320868</parentid>
      <timestamp>2025-07-07T06:29:42Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply</comment>
      <origin>2320870</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="142950" sha1="d8p3ectyayrjbhukb0vnwahrk3zke0b" xml:space="preserve">== 2025-07-07 ==
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>d8p3ectyayrjbhukb0vnwahrk3zke0b</sha1>
    </revision>
    <revision>
      <id>2320874</id>
      <parentid>2320870</parentid>
      <timestamp>2025-07-07T07:04:47Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - T398720</comment>
      <origin>2320874</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143041" sha1="2qctimndpd0f3k5tocnzl6fyn9bxh36" xml:space="preserve">== 2025-07-07 ==
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>2qctimndpd0f3k5tocnzl6fyn9bxh36</sha1>
    </revision>
    <revision>
      <id>2320875</id>
      <parentid>2320874</parentid>
      <timestamp>2025-07-07T07:10:07Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply</comment>
      <origin>2320875</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143152" sha1="2g38emit0vfn35wotct8wolth5vjzk2" xml:space="preserve">== 2025-07-07 ==
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>2g38emit0vfn35wotct8wolth5vjzk2</sha1>
    </revision>
    <revision>
      <id>2320876</id>
      <parentid>2320875</parentid>
      <timestamp>2025-07-07T07:11:05Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply</comment>
      <origin>2320876</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143262" sha1="s10jbdm715mcmjqv2mht9r0lmc2kqw2" xml:space="preserve">== 2025-07-07 ==
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>s10jbdm715mcmjqv2mht9r0lmc2kqw2</sha1>
    </revision>
    <revision>
      <id>2320877</id>
      <parentid>2320876</parentid>
      <timestamp>2025-07-07T07:13:57Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 T397612</comment>
      <origin>2320877</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143432" sha1="1eo9zeozcs18w0ib4pu6932kg751slc" xml:space="preserve">== 2025-07-07 ==
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>1eo9zeozcs18w0ib4pu6932kg751slc</sha1>
    </revision>
    <revision>
      <id>2320878</id>
      <parentid>2320877</parentid>
      <timestamp>2025-07-07T07:22:00Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 T397612', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json</comment>
      <origin>2320878</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143677" sha1="r9hjclohusd5o9nwm0hzcifu0qpxap4" xml:space="preserve">== 2025-07-07 ==
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>r9hjclohusd5o9nwm0hzcifu0qpxap4</sha1>
    </revision>
    <revision>
      <id>2320880</id>
      <parentid>2320878</parentid>
      <timestamp>2025-07-07T07:25:22Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui: Starting x1 eqiad failover from db1237 to db1220 - T397612</comment>
      <origin>2320880</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143773" sha1="erf5zoegzo94iqeaytli0zszczri893" xml:space="preserve">== 2025-07-07 ==
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>erf5zoegzo94iqeaytli0zszczri893</sha1>
    </revision>
    <revision>
      <id>2320881</id>
      <parentid>2320880</parentid>
      <timestamp>2025-07-07T07:25:46Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>vgutierrez: depooling cp7006 to test Ia82b9354a5b9e7bd5443b4af0888325919ddb19e - T397917</comment>
      <origin>2320881</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143898" sha1="b40z0fdil3khoohfduluial1sd4jsmr" xml:space="preserve">== 2025-07-07 ==
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>b40z0fdil3khoohfduluial1sd4jsmr</sha1>
    </revision>
    <revision>
      <id>2320882</id>
      <parentid>2320881</parentid>
      <timestamp>2025-07-07T07:50:21Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@dns1006: START - running authdns-update</comment>
      <origin>2320882</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="143957" sha1="qlpi5l3c0i1axtuz0zhdu50jei1hyq1" xml:space="preserve">== 2025-07-07 ==
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>qlpi5l3c0i1axtuz0zhdu50jei1hyq1</sha1>
    </revision>
    <revision>
      <id>2320883</id>
      <parentid>2320882</parentid>
      <timestamp>2025-07-07T07:51:26Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@dns1006: END - running authdns-update</comment>
      <origin>2320883</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="144014" sha1="ginjni1acj8pyazsu5dx4zhseidgx0k" xml:space="preserve">== 2025-07-07 ==
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>ginjni1acj8pyazsu5dx4zhseidgx0k</sha1>
    </revision>
    <revision>
      <id>2320884</id>
      <parentid>2320883</parentid>
      <timestamp>2025-07-07T07:52:58Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write T397612', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json</comment>
      <origin>2320884</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="144290" sha1="m8mgzhmceec1tzn9qpankpdotbc3esf" xml:space="preserve">== 2025-07-07 ==
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>m8mgzhmceec1tzn9qpankpdotbc3esf</sha1>
    </revision>
    <revision>
      <id>2320885</id>
      <parentid>2320884</parentid>
      <timestamp>2025-07-07T07:53:10Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 T397612', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json</comment>
      <origin>2320885</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="144524" sha1="3uerye6e8hada36imgnd1dvfv1a2wzu" xml:space="preserve">== 2025-07-07 ==
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>3uerye6e8hada36imgnd1dvfv1a2wzu</sha1>
    </revision>
    <revision>
      <id>2320886</id>
      <parentid>2320885</parentid>
      <timestamp>2025-07-07T07:53:24Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>vgutierrez: repooling cp7006 with Ia82b9354a5b9e7bd5443b4af0888325919ddb19e applied - T397917</comment>
      <origin>2320886</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="144654" sha1="1kdvm1dn9pkdalee6qb7c5s7b5l9hma" xml:space="preserve">== 2025-07-07 ==
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>1kdvm1dn9pkdalee6qb7c5s7b5l9hma</sha1>
    </revision>
    <revision>
      <id>2320887</id>
      <parentid>2320886</parentid>
      <timestamp>2025-07-07T08:00:10Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance</comment>
      <origin>2320887</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="144799" sha1="fl5ofzq40nscbp8qn10qo941vie13y4" xml:space="preserve">== 2025-07-07 ==
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>fl5ofzq40nscbp8qn10qo941vie13y4</sha1>
    </revision>
    <revision>
      <id>2320888</id>
      <parentid>2320887</parentid>
      <timestamp>2025-07-07T08:00:57Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;</comment>
      <origin>2320888</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="145007" sha1="7rsnf1vx8ouqwgwd8vcy0aaukyg01ii" xml:space="preserve">== 2025-07-07 ==
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>7rsnf1vx8ouqwgwd8vcy0aaukyg01ii</sha1>
    </revision>
    <revision>
      <id>2320889</id>
      <parentid>2320888</parentid>
      <timestamp>2025-07-07T08:00:59Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003</comment>
      <origin>2320889</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="145214" sha1="l44e9ty1ickj7ga4agmflwbvq6t9lo4" xml:space="preserve">== 2025-07-07 ==
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>l44e9ty1ickj7ga4agmflwbvq6t9lo4</sha1>
    </revision>
    <revision>
      <id>2320890</id>
      <parentid>2320889</parentid>
      <timestamp>2025-07-07T08:01:30Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003</comment>
      <origin>2320890</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="145440" sha1="ejf2bd8uwf4t4xxi0kj0wvsf5hsnufu" xml:space="preserve">== 2025-07-07 ==
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>ejf2bd8uwf4t4xxi0kj0wvsf5hsnufu</sha1>
    </revision>
    <revision>
      <id>2320891</id>
      <parentid>2320890</parentid>
      <timestamp>2025-07-07T08:01:31Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;</comment>
      <origin>2320891</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="145667" sha1="qgegg8hzoykkv4266pw3naqnj9sr5q4" xml:space="preserve">== 2025-07-07 ==
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>qgegg8hzoykkv4266pw3naqnj9sr5q4</sha1>
    </revision>
    <revision>
      <id>2320892</id>
      <parentid>2320891</parentid>
      <timestamp>2025-07-07T08:15:39Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance</comment>
      <origin>2320892</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="145819" sha1="gwnerdkz0ffhg5v1ae2eo32t62jp710" xml:space="preserve">== 2025-07-07 ==
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>gwnerdkz0ffhg5v1ae2eo32t62jp710</sha1>
    </revision>
    <revision>
      <id>2320899</id>
      <parentid>2320892</parentid>
      <timestamp>2025-07-07T09:06:42Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance</comment>
      <origin>2320899</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146002" sha1="rynjels51s5nk744guh5phcwxxubcoz" xml:space="preserve">== 2025-07-07 ==
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>rynjels51s5nk744guh5phcwxxubcoz</sha1>
    </revision>
    <revision>
      <id>2320900</id>
      <parentid>2320899</parentid>
      <timestamp>2025-07-07T09:09:14Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320900</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146113" sha1="69vd2eqqlp4jypc6qfx4acy5t4ikyg7" xml:space="preserve">== 2025-07-07 ==
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>69vd2eqqlp4jypc6qfx4acy5t4ikyg7</sha1>
    </revision>
    <revision>
      <id>2320901</id>
      <parentid>2320900</parentid>
      <timestamp>2025-07-07T09:13:54Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui: Failover m2 from db1250 to db1228 - T397633</comment>
      <origin>2320901</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146194" sha1="eil4r0o7d6x5moowl1c8p4vpp823oyq" xml:space="preserve">== 2025-07-07 ==
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>eil4r0o7d6x5moowl1c8p4vpp823oyq</sha1>
    </revision>
    <revision>
      <id>2320902</id>
      <parentid>2320901</parentid>
      <timestamp>2025-07-07T09:18:42Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320902</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146326" sha1="j7bhelmtlk5ddbpr5zc1ypuy5uoa6ga" xml:space="preserve">== 2025-07-07 ==
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>j7bhelmtlk5ddbpr5zc1ypuy5uoa6ga</sha1>
    </revision>
    <revision>
      <id>2320903</id>
      <parentid>2320902</parentid>
      <timestamp>2025-07-07T09:21:38Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance</comment>
      <origin>2320903</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146471" sha1="bct91mnk02ryfm7qgg17ex113w5cptf" xml:space="preserve">== 2025-07-07 ==
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>bct91mnk02ryfm7qgg17ex113w5cptf</sha1>
    </revision>
    <revision>
      <id>2320904</id>
      <parentid>2320903</parentid>
      <timestamp>2025-07-07T09:25:05Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320904</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146582" sha1="oiglnxv8x5dm0l01esnr4aw9yqw07ws" xml:space="preserve">== 2025-07-07 ==
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>oiglnxv8x5dm0l01esnr4aw9yqw07ws</sha1>
    </revision>
    <revision>
      <id>2320905</id>
      <parentid>2320904</parentid>
      <timestamp>2025-07-07T09:42:21Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320905</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146714" sha1="ov8tcp5a6ja6n090m318m85ehv0ax2i" xml:space="preserve">== 2025-07-07 ==
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>ov8tcp5a6ja6n090m318m85ehv0ax2i</sha1>
    </revision>
    <revision>
      <id>2320906</id>
      <parentid>2320905</parentid>
      <timestamp>2025-07-07T09:43:53Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320906</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146825" sha1="42is5wg2mcg4yv1i719n5aiwotwknx8" xml:space="preserve">== 2025-07-07 ==
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>42is5wg2mcg4yv1i719n5aiwotwknx8</sha1>
    </revision>
    <revision>
      <id>2320907</id>
      <parentid>2320906</parentid>
      <timestamp>2025-07-07T09:58:10Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320907</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="146957" sha1="8m9klr90dnhz8fqn7y486xv76hn328c" xml:space="preserve">== 2025-07-07 ==
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>8m9klr90dnhz8fqn7y486xv76hn328c</sha1>
    </revision>
    <revision>
      <id>2320908</id>
      <parentid>2320907</parentid>
      <timestamp>2025-07-07T10:09:22Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe</comment>
      <origin>2320908</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147088" sha1="k0rk6xel07wpp3q2jp0rmjkfzos18l7" xml:space="preserve">== 2025-07-07 ==
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>k0rk6xel07wpp3q2jp0rmjkfzos18l7</sha1>
    </revision>
    <revision>
      <id>2320909</id>
      <parentid>2320908</parentid>
      <timestamp>2025-07-07T10:13:17Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe</comment>
      <origin>2320909</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147238" sha1="i2pk1zpfwj3dx1vfspb8b2b6bbu6ecz" xml:space="preserve">== 2025-07-07 ==
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>i2pk1zpfwj3dx1vfspb8b2b6bbu6ecz</sha1>
    </revision>
    <revision>
      <id>2320910</id>
      <parentid>2320909</parentid>
      <timestamp>2025-07-07T10:17:16Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320910</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147349" sha1="q1lloefsdtzzg4057o24nxnh7zg5v6n" xml:space="preserve">== 2025-07-07 ==
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>q1lloefsdtzzg4057o24nxnh7zg5v6n</sha1>
    </revision>
    <revision>
      <id>2320911</id>
      <parentid>2320910</parentid>
      <timestamp>2025-07-07T10:24:09Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 T335491</comment>
      <origin>2320911</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147476" sha1="9okhri15yycb1rco7lvdvy562lwy43d" xml:space="preserve">== 2025-07-07 ==
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>9okhri15yycb1rco7lvdvy562lwy43d</sha1>
    </revision>
    <revision>
      <id>2320912</id>
      <parentid>2320911</parentid>
      <timestamp>2025-07-07T10:30:58Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320912</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147608" sha1="pcrqyb242usahsbvwfy7t577oyzjtvz" xml:space="preserve">== 2025-07-07 ==
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>pcrqyb242usahsbvwfy7t577oyzjtvz</sha1>
    </revision>
    <revision>
      <id>2320913</id>
      <parentid>2320912</parentid>
      <timestamp>2025-07-07T10:35:59Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320913</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147719" sha1="ql6yk3w4g9kxk7772q887qlaol0qgwn" xml:space="preserve">== 2025-07-07 ==
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>ql6yk3w4g9kxk7772q887qlaol0qgwn</sha1>
    </revision>
    <revision>
      <id>2320914</id>
      <parentid>2320913</parentid>
      <timestamp>2025-07-07T10:45:00Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance</comment>
      <origin>2320914</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="147871" sha1="la17rvkit5j8k5i3o8qu6wpow9azrxp" xml:space="preserve">== 2025-07-07 ==
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>la17rvkit5j8k5i3o8qu6wpow9azrxp</sha1>
    </revision>
    <revision>
      <id>2320915</id>
      <parentid>2320914</parentid>
      <timestamp>2025-07-07T10:45:16Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm</comment>
      <origin>2320915</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148003" sha1="otufpfy736uwwi74g80i3aiayqzkms2" xml:space="preserve">== 2025-07-07 ==
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>otufpfy736uwwi74g80i3aiayqzkms2</sha1>
    </revision>
    <revision>
      <id>2320916</id>
      <parentid>2320915</parentid>
      <timestamp>2025-07-07T10:53:48Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance</comment>
      <origin>2320916</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148148" sha1="hipgosqqa2gtu3so33h998dzndm9ar0" xml:space="preserve">== 2025-07-07 ==
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>hipgosqqa2gtu3so33h998dzndm9ar0</sha1>
    </revision>
    <revision>
      <id>2320917</id>
      <parentid>2320916</parentid>
      <timestamp>2025-07-07T11:02:35Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>moritzm: installing modsecurity-apache security updates</comment>
      <origin>2320917</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148212" sha1="87tvomm3sbiu3m8xo2i60voui56bucc" xml:space="preserve">== 2025-07-07 ==
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>87tvomm3sbiu3m8xo2i60voui56bucc</sha1>
    </revision>
    <revision>
      <id>2320918</id>
      <parentid>2320917</parentid>
      <timestamp>2025-07-07T11:06:00Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance</comment>
      <origin>2320918</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148364" sha1="g5qdtyqfbao02mzfi43r9lcvckjf9rm" xml:space="preserve">== 2025-07-07 ==
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>g5qdtyqfbao02mzfi43r9lcvckjf9rm</sha1>
    </revision>
    <revision>
      <id>2320919</id>
      <parentid>2320918</parentid>
      <timestamp>2025-07-07T11:06:23Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe</comment>
      <origin>2320919</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148495" sha1="ptjsp1bnprjyawqonz64ecwdxjsmhmu" xml:space="preserve">== 2025-07-07 ==
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>ptjsp1bnprjyawqonz64ecwdxjsmhmu</sha1>
    </revision>
    <revision>
      <id>2320921</id>
      <parentid>2320919</parentid>
      <timestamp>2025-07-07T11:10:17Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe</comment>
      <origin>2320921</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148645" sha1="9mmeoy7oxqvlepbm27w9ulbbhv1aiis" xml:space="preserve">== 2025-07-07 ==
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>9mmeoy7oxqvlepbm27w9ulbbhv1aiis</sha1>
    </revision>
    <revision>
      <id>2320928</id>
      <parentid>2320921</parentid>
      <timestamp>2025-07-07T11:25:07Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169|Revert^2 &quot;Clean up EventBus and jobs config&quot;]]</comment>
      <origin>2320928</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148781" sha1="lqdzdhjj0cnknkujqrt46gckisifqsq" xml:space="preserve">== 2025-07-07 ==
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>lqdzdhjj0cnknkujqrt46gckisifqsq</sha1>
    </revision>
    <revision>
      <id>2320929</id>
      <parentid>2320928</parentid>
      <timestamp>2025-07-07T11:42:20Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie</comment>
      <origin>2320929</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="148890" sha1="7flo6764iqax1k9rmrmbmenwm3hxg90" xml:space="preserve">== 2025-07-07 ==
* 11:42 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>7flo6764iqax1k9rmrmbmenwm3hxg90</sha1>
    </revision>
    <revision>
      <id>2320930</id>
      <parentid>2320929</parentid>
      <timestamp>2025-07-07T11:46:22Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1165169|Revert^2 &quot;Clean up EventBus and jobs config&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.</comment>
      <origin>2320930</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="149124" sha1="1dpn97qa9ggrkqymqyz71fbpkmv8rsx" xml:space="preserve">== 2025-07-07 ==
* 11:46 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:42 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>1dpn97qa9ggrkqymqyz71fbpkmv8rsx</sha1>
    </revision>
    <revision>
      <id>2320931</id>
      <parentid>2320930</parentid>
      <timestamp>2025-07-07T11:47:24Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>ladsgroup@deploy1003: ladsgroup: Continuing with sync</comment>
      <origin>2320931</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="149186" sha1="57q46ofjsmhow4tto366wxaots2al4d" xml:space="preserve">== 2025-07-07 ==
* 11:47 ladsgroup@deploy1003: ladsgroup: Continuing with sync
* 11:46 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:42 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>57q46ofjsmhow4tto366wxaots2al4d</sha1>
    </revision>
    <revision>
      <id>2320932</id>
      <parentid>2320931</parentid>
      <timestamp>2025-07-07T11:51:02Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd</comment>
      <origin>2320932</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="149304" sha1="1246bpt7ricnkjazhetggdfg2yo1ab6" xml:space="preserve">== 2025-07-07 ==
* 11:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 11:47 ladsgroup@deploy1003: ladsgroup: Continuing with sync
* 11:46 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:42 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>1246bpt7ricnkjazhetggdfg2yo1ab6</sha1>
    </revision>
    <revision>
      <id>2320933</id>
      <parentid>2320932</parentid>
      <timestamp>2025-07-07T11:54:59Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool db2146 T398433', diff saved to https://phabricator.wikimedia.org/P78771 and previous config saved to /var/cache/conftool/dbconfig/20250707-115457-ladsgroup.json</comment>
      <origin>2320933</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="149542" sha1="3pifao4m378pm5bfq5hw3aaynixm150" xml:space="preserve">== 2025-07-07 ==
* 11:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool db2146 [[phab:T398433|T398433]]', diff saved to https://phabricator.wikimedia.org/P78771 and previous config saved to /var/cache/conftool/dbconfig/20250707-115457-ladsgroup.json
* 11:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 11:47 ladsgroup@deploy1003: ladsgroup: Continuing with sync
* 11:46 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:42 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>3pifao4m378pm5bfq5hw3aaynixm150</sha1>
    </revision>
    <revision>
      <id>2320934</id>
      <parentid>2320933</parentid>
      <timestamp>2025-07-07T11:56:24Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS trixie</comment>
      <origin>2320934</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="149672" sha1="fp2d1shwzkhxi2r2cb04hm6djvatnxm" xml:space="preserve">== 2025-07-07 ==
* 11:56 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS trixie
* 11:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool db2146 [[phab:T398433|T398433]]', diff saved to https://phabricator.wikimedia.org/P78771 and previous config saved to /var/cache/conftool/dbconfig/20250707-115457-ladsgroup.json
* 11:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 11:47 ladsgroup@deploy1003: ladsgroup: Continuing with sync
* 11:46 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:42 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie
* 11:25 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1165169{{!}}Revert^2 &quot;Clean up EventBus and jobs config&quot;]]
* 11:10 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 11:06 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 11:02 moritzm: installing modsecurity-apache security updates
* 10:53 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 10:45 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance
* 10:35 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:30 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:24 Emperor: remove swift-account-stats_machinetranslation:prod time &amp; service from thanos-fe1004 [[phab:T335491|T335491]]
* 10:17 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 10:09 root@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 09:58 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:43 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:42 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:25 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance
* 09:18 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 09:13 marostegui: Failover m2 from db1250 to db1228 - [[phab:T397633|T397633]]
* 09:09 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance
* 08:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:01 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003
* 08:00 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: logging of deny actions; add rename functionality - oblivian@cumin1003&quot;
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance
* 07:53 vgutierrez: repooling cp7006 with {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} applied - [[phab:T397917|T397917]]
* 07:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1237 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json
* 07:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json
* 07:51 marostegui@dns1006: END - running authdns-update
* 07:50 marostegui@dns1006: START - running authdns-update
* 07:25 vgutierrez: depooling cp7006 to test {{Gerrit|Ia82b9354a5b9e7bd5443b4af0888325919ddb19e}} - [[phab:T397917|T397917]]
* 07:25 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T397612|T397612]]
* 07:22 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T397612|T397612]]', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 [[phab:T397612|T397612]]
* 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 07:04 vgutierrez: testing haproxy 2.8.15 in cp5017 and cp5025 - [[phab:T398720|T398720]]
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 06:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply

== 2025-07-04 ==
* 21:39 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s)
* 21:33 krinkle@deploy1003: krinkle: Continuing with sync
* 21:23 krinkle@deploy1003: krinkle: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:21 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1166438{{!}}beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]]
* 20:32 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s)
* 20:26 krinkle@deploy1003: krinkle: Continuing with sync
* 18:59 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165989{{!}}beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999{{!}}beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]]
* 15:14 vgutierrez: fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o)
* 14:46 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye
* 14:40 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:36 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 14:29 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:20 vgutierrez: repooling cp7006
* 14:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 14:12 vgutierrez: depooling cp7006 for testing purposes
* 14:09 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 14:06 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye
* 14:01 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 13:15 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 13:08 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:59 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 12:51 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 12:31 vgutierrez: repool cp7006
* 12:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 12:31 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 12:11 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s)
* 12:11 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0
* 11:08 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 11:05 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003
* 10:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance
* 10:51 elukey@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm
* 10:43 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance
* 10:41 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
* 10:27 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s)
* 10:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 10:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 10:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 10:18 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 10:01 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backupmon1001.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 09:07 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: Maintenance and reboot
* 08:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:48 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet
* 08:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 08:37 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:32 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6001.drmrs.wmnet to cluster drmrs01 and group B12
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2020.codfw.wmnet
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6001.drmrs.wmnet with OS bookworm
* 08:03 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:58 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:56 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 07:53 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2020.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2019.codfw.wmnet
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti2019.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:42 vgutierrez: depooling cp7006 for testing purposes
* 07:42 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:39 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6001.drmrs.wmnet with reason: host reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts ganeti2019.codfw.wmnet
* 07:21 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6001.drmrs.wmnet with OS bookworm
* 07:19 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6001.drmrs.wmnet with reason: reimage
* 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2003.codfw.wmnet
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2003.codfw.wmnet
* 06:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 06:32 moritzm: failover Ganeti master in drmrs01 to ganeti6003 [[phab:T382513|T382513]]
* 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 06:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 06:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 06:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 06:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 06:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 06:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 04:32 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1042.eqiad.wmnet with OS bullseye
* 04:25 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:51 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:46 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 03:21 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1042
* 03:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1042
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 03:19 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:19 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudcephosd1042] - vriley@cumin1002&quot;
* 03:15 vriley@cumin1002: START - Cookbook sre.dns.netbox

== 2025-07-03 ==
* 21:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 21:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 21:18 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet
* 21:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet
* 21:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to drbd
* 21:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] (duration: 08m 37s)
* 21:11 zabe@deploy1003: kharlan, zabe: Continuing with sync
* 21:09 zabe@deploy1003: kharlan, zabe: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166178{{!}}special: Do not throw ErrorPageError from getRedirect() (T398167)]], [[gerrit:1166264{{!}}Set categorylinks to read new on small wikis (T397912)]]
* 20:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 20:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to drbd
* 20:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 20:47 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] (duration: 08m 27s)
* 20:41 arlolra@deploy1003: arlolra, matmarex: Continuing with sync
* 20:40 arlolra@deploy1003: arlolra, matmarex: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1166236{{!}}Use FallbackContentHandler for undeployed JsonConfig content handlers (T124748)]], [[gerrit:1166012{{!}}ExtensionDistributor: Mark 1.44 as stable; remove 1.42 as EOL (T390798 T389313)]]
* 20:36 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] (duration: 08m 30s)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:29 cscott@deploy1003: cscott: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1166206{{!}}skin: Omit &quot;rendered with&quot; phrase when the message is disabled (T398616)]]
* 20:12 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] (duration: 08m 32s)
* 20:06 zabe@deploy1003: zabe: Continuing with sync
* 20:05 zabe@deploy1003: zabe: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1166133{{!}}Use correct index on categorylinks (T385890)]]
* 19:36 joal@deploy1003: Finished deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics (duration: 01m 02s)
* 19:35 joal@deploy1003: Started deploy [airflow-dags/analytics@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics
* 19:34 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test (duration: 00m 16s)
* 19:34 joal@deploy1003: Started deploy [airflow-dags/analytics_test@7ba4a7b]: BUGFIX - Synchronize artifact for airflow_dags/analytics_test
* 17:33 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
* 17:26 joal@deploy1003: Finished deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics (duration: 00m 40s)
* 17:25 joal@deploy1003: Started deploy [airflow-dags/analytics@9088e59]: Synchronize artifacts for airflow_dags/analytics
* 17:24 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test (duration: 00m 15s)
* 17:24 joal@deploy1003: Started deploy [airflow-dags/analytics_test@9088e59]: Synchronize artifacat for airflow_dags/analytics_test
* 17:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:15 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 17:13 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 17:12 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 17:01 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 16:32 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 16:32 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 16:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 16:11 vgutierrez: repooling cp7006
* 16:09 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 16:09 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 15:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 15:52 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 15:46 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 15:42 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:38 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
* 15:34 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: testing
* 15:33 vgutierrez: depooling cp7006 for testing
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78755 and previous config saved to /var/cache/conftool/dbconfig/20250703-153141-fceratto.json
* 15:25 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
* 15:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:22 vgutierrez: lvs5006 migrated to katran - [[phab:T396561|T396561]]
* 15:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5006.eqsin.wmnet
* 15:21 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5006.eqsin.wmnet
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78754 and previous config saved to /var/cache/conftool/dbconfig/20250703-151633-fceratto.json
* 15:10 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5006.eqsin.wmnet with reason: katran migration
* 15:04 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw
* 15:04 jmm@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P78753 and previous config saved to /var/cache/conftool/dbconfig/20250703-150126-fceratto.json
* 14:56 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1007.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet
* 14:51 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet
* 14:50 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1006.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 14:50 volans: uploaded debmonitor-server,python3-debmonitor_0.6.6 to apt.wikimedia.org bookworm-wikimedia
* 14:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet
* 14:48 vgutierrez: repooling cp7006
* 14:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78752 and previous config saved to /var/cache/conftool/dbconfig/20250703-144619-fceratto.json
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet
* 14:45 jmm@dns1004: END - running authdns-update
* 14:44 jmm@dns1004: START - running authdns-update
* 14:43 jmm@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
* 14:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P78751 and previous config saved to /var/cache/conftool/dbconfig/20250703-143854-fceratto.json
* 14:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 14:32 moritzm: installing bootstrap4 security updates
* 14:23 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
* 14:17 vgutierrez: depooling cp7006 for testing
* 14:09 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1007.eqiad.wmnet with reason: Maintenance and reboot
* 14:08 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1006.eqiad.wmnet with reason: Maintenance and reboot
* 14:05 moritzm: restarting clamav to pick up libxml security updates
* 14:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet
* 13:59 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet
* 13:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet
* 13:46 sukhe: sudo cumin 'A:wikidough' &quot;disable-puppet 'merging CR 1163859'&quot;
* 13:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet
* 13:40 moritzm: installing libxml2 security updates on bookworm
* 13:40 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet
* 13:40 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet
* 13:39 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 13:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet
* 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to drbd
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to drbd
* 13:22 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:21 sukhe: sudo cumin -b11 'C:bird' &quot;run-puppet-agent --enable 'merging CR 1163858'&quot;: NOOP change [[phab:T374619|T374619]]
* 13:20 TheresNoTime: done UTC afternoon backport window
* 13:18 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] (duration: 14m 04s)
* 13:18 sukhe: sudo cumin 'C:bird' &quot;disable-puppet 'merging CR 1163858'&quot;: [[phab:T374619|T374619]]
* 13:17 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to drbd
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1166155{{!}}InitialiseSettings: Enable wgTemplateDataEnableDiscovery as default (T377978)]], [[gerrit:1165635{{!}}Allow abusefilter block action on plwikiquote (T398137)]]
* 12:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to drbd
* 12:59 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 12:54 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0 (duration: 03m 20s)
* 12:51 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@09893e3]: bump section topics to v1.7.0
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to drbd
* 11:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 11:45 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to drbd
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1005.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:35 jiji@deploy1003: Finished scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production (duration: 06m 59s)
* 11:30 jiji@deploy1003: Started scap sync-world: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 11:29 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6003.drmrs.wmnet to cluster drmrs01 and group B12
* 11:27 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys (duration: 44m 16s)
* 11:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:21 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1004.eqiad.wmnet: Renew puppet certificate - jynus@cumin1002
* 11:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:15 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:15 effie: starting staged rollout of  Excimer to 1.2.5,  mw-api-ext
* 11:15 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:11 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003&quot;
* 11:07 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:05 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:05 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
* 11:04 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:03 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 10:51 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:50 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 10:49 effie: starting staged rollout of  Excimer to 1.2.5  mw-debug first, mw-api-int next
* 10:47 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 10:43 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production in progress, blocking deploys
* 10:42 jiji@deploy1003: Stopping before sync operations
* 10:26 jiji@deploy1003: Started scap sync-world:  [[phab:T397907|T397907]] - Upgrade Excimer to 1.2.5 in production
* 10:23 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1005.eqiad.wmnet with reason: Maintenance and reboot
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 10:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 10:05 volans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on debmonitor2003.codfw.wmnet,debmonitor1003.eqiad.wmnet,debmonitor-dev2001.codfw.wmnet with reason: deploy new version
* 10:00 volans: upgrading production debmonitor-server to the latest v0.6.5
* 09:39 fceratto@cumin1002: dbctl commit (dc=all): 'Set db2213 weights [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78747 and previous config saved to /var/cache/conftool/dbconfig/20250703-093943-fceratto.json
* 09:36 fceratto@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78746 and previous config saved to /var/cache/conftool/dbconfig/20250703-093612-fceratto.json
* 09:34 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T398594|T398594]]
* 09:31 vgutierrez: repooling cp7006
* 09:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet
* 09:25 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2192 from API/vslow/dump [[phab:T398594|T398594]]', diff saved to https://phabricator.wikimedia.org/P78745 and previous config saved to /var/cache/conftool/dbconfig/20250703-092522-fceratto.json
* 09:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398594|T398594]]
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1004.eqiad.wmnet with reason: Maintenance and reboot
* 09:21 fceratto@cumin1002: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T398593|T398593]]
* 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet
* 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6003.drmrs.wmnet with reason: host reimage
* 08:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet
* 08:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet
* 08:37 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet
* 08:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6003.drmrs.wmnet with OS bookworm
* 08:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6003.drmrs.wmnet with reason: reimage
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of bast6003.wikimedia.org to plain
* 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6001.drmrs.wmnet to plain
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6001.drmrs.wmnet to plain
* 08:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6001.drmrs.wmnet to plain
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6001.wikimedia.org to plain
* 08:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6001.wikimedia.org to plain
* 08:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:14 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:13 volans: uploaded debmonitor-server,python3-debmonitor_0.6.5 to apt.wikimedia.org bookworm-wikimedia
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install6002.wikimedia.org to plain
* 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install6002.wikimedia.org to plain
* 08:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:53 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 07:53 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78744 and previous config saved to /var/cache/conftool/dbconfig/20250703-075225-ladsgroup.json
* 07:52 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 07:51 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 07:50 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 07:49 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 07:42 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7006.magru.wmnet with reason: haproxy testing
* 07:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:39 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: search in response reasons - oblivian@cumin1003
* 07:38 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Feature: search in response reasons - oblivian@cumin1003&quot;
* 07:34 effie: upload php-excimer_1.2.5-1+wmf11u1
* 07:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] (duration: 12m 16s)
* 07:21 ladsgroup@deploy1003: musikanimal, ladsgroup: Continuing with sync
* 07:18 vgutierrez: depooling cp7006 for requestctl debugging
* 07:16 ladsgroup@deploy1003: musikanimal, ladsgroup: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1166067{{!}}codeFolding: fix folding &lt;ref&gt; (T398430)]]
* 07:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 07:02 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on prometheus6002.drmrs.wmnet with reason: switch disk type back to DRBD
* 07:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 06:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet
* 06:49 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet
* 06:47 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 06:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 03:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 03:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:18 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 03:06 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:56 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:55 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:54 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:53 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 00:03 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - [[phab:T398245|T398245]]

== 2025-07-02 ==
* 23:40 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:38 tzatziki: removing 15 files for legal compliance
* 23:25 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 23:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:07 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 23:05 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 23:02 ryankemper: [WDQS] `ryankemper@wdqs2009:~$ sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service`
* 22:51 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:51 dancy@deploy1003: Installation of scap version &quot;4.186.0&quot; completed for 2 hosts
* 22:49 dancy@deploy1003: Installing scap version &quot;4.186.0&quot; for 2 host(s)
* 22:49 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:40 ryankemper: [WDQS] Restart wdqs-blazegraph on wdqs2009
* 22:27 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)
* 22:21 zabe@deploy1003: zabe: Continuing with sync
* 22:21 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 22:19 zabe@deploy1003: zabe: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:19 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165996{{!}}ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]
* 22:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:59 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:55 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:23 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:23 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:22 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:20 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:16 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 21:15 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 21:14 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 21:13 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 21:12 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 21:11 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:05 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] (duration: 29m 54s)
* 20:59 krinkle@deploy1003: krinkle: Continuing with sync
* 20:37 krinkle@deploy1003: krinkle: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:35 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1165983{{!}}missing.php: Support beta suffix for auth.wikimedia error page (T289318)]]
* 20:34 swfrench-wmf: reprepro include dh-php_5.5+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:31 krinkle@deploy1003: Finished scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}} (duration: 03m 06s)
* 20:30 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:29 swfrench-wmf: reprepro include php-defaults_94+wmf11u1 in component/php83 - [[phab:T398245|T398245]]
* 20:28 krinkle@deploy1003: Started scap sync-world: Beta patches {{Gerrit|Iff58893f}}, {{Gerrit|I62b31535}}, {{Gerrit|I228d7766a57}}
* 20:10 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:06 Krinkle: krinkle@deploy1003:/srv/mediawiki$ git remote rm gerrit -- Fix `jforrester@gerrit.wikimedia.org: Permission denied (publickey).` There were two remotes: $ git remote -v gerrit  ssh://jforrester@gerrit origin  ssh://gerrit.wikimedia.org:29418
* 20:06 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:47 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 18:42 swfrench-wmf: reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83  - [[phab:T398245|T398245]]
* 17:53 swfrench-wmf: reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - [[phab:T398245|T398245]]
* 17:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2330.codfw.wmnet
* 17:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2329.codfw.wmnet
* 17:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2328.codfw.wmnet
* 17:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2327.codfw.wmnet
* 17:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2326.codfw.wmnet
* 17:29 dzahn@dns1004: END - running authdns-update
* 17:28 dzahn@dns1004: START - running authdns-update
* 17:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2326.codfw.wmnet
* 17:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2325.codfw.wmnet
* 17:21 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2324.codfw.wmnet
* 17:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2323.codfw.wmnet
* 17:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2322.codfw.wmnet
* 17:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2321.codfw.wmnet
* 16:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2320.codfw.wmnet
* 16:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2319.codfw.wmnet
* 16:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2318.codfw.wmnet
* 16:47 inflatador: bking@cumin1002 restarting cirrrussearch codfw [[phab:T397227|T397227]]
* 16:44 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 16:43 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2318.codfw.wmnet
* 16:43 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2317.codfw.wmnet
* 16:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 16:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2317.codfw.wmnet
* 16:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2316.codfw.wmnet
* 16:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2315.codfw.wmnet
* 16:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2314.codfw.wmnet
* 16:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2313.codfw.wmnet
* 16:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2312.codfw.wmnet
* 16:13 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2312.codfw.wmnet
* 16:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2311.codfw.wmnet
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply
* 16:08 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2311.codfw.wmnet
* 16:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2310.codfw.wmnet
* 16:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2309.codfw.wmnet
* 15:56 vgutierrez: switch lvs4010 to katran - 10.128.0.11
* 15:56 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2309.codfw.wmnet
* 15:56 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2308.codfw.wmnet
* 15:55 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] (duration: 08m 51s)
* 15:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2308.codfw.wmnet
* 15:53 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2307.codfw.wmnet
* 15:49 jnuche@deploy1003: jnuche, daimona: Continuing with sync
* 15:49 jnuche@deploy1003: jnuche, daimona: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2307.codfw.wmnet
* 15:48 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2306.codfw.wmnet
* 15:47 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4010.ulsfo.wmnet with reason: katran migration
* 15:46 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1165894{{!}}Rename EventRegistration::$meetingAddress to $address for cache compat (T398413)]]
* 15:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2306.codfw.wmnet
* 15:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2305.codfw.wmnet
* 15:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2304.codfw.wmnet
* 15:33 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2303.codfw.wmnet
* 15:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2303.codfw.wmnet
* 15:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2302.codfw.wmnet
* 15:22 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2301.codfw.wmnet
* 15:20 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 15:17 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2301.codfw.wmnet
* 15:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2300.codfw.wmnet
* 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:15 vgutierrez: repool cp7006
* 15:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 15:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 15:13 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2300.codfw.wmnet
* 15:11 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2299.codfw.wmnet
* 15:11 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:08 dancy@deploy1003: Installation of scap version &quot;4.185.0&quot; completed for 2 hosts
* 15:06 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 15:06 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2299.codfw.wmnet
* 15:06 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2298.codfw.wmnet
* 15:06 dancy@deploy1003: Installing scap version &quot;4.185.0&quot; for 2 host(s)
* 15:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:03 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6002.drmrs.wmnet to cluster drmrs02 and group B13
* 15:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
* 15:01 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2298.codfw.wmnet
* 15:01 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2297.codfw.wmnet
* 15:00 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:57 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2014
* 14:56 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2014
* 14:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2297.codfw.wmnet
* 14:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2296.codfw.wmnet
* 14:55 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
* 14:52 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2296.codfw.wmnet
* 14:50 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2295.codfw.wmnet
* 14:47 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
* 14:45 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2295.codfw.wmnet
* 14:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2294.codfw.wmnet
* 14:42 godog: bounce thanos-store on titan1002
* 14:40 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] (duration: 08m 26s)
* 14:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2294.codfw.wmnet
* 14:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2293.codfw.wmnet
* 14:39 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0 (duration: 00m 47s)
* 14:38 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@1bb179b]: bump section topics to v1.6.0
* 14:38 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:38 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 14:35 oblivian@deploy1003: zabe, oblivian: Continuing with sync
* 14:34 oblivian@deploy1003: zabe, oblivian: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2293.codfw.wmnet
* 14:34 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2292.codfw.wmnet
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:31 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1165897{{!}}Revert &quot;group1: Set categorylinks to read new&quot;]]
* 14:31 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet
* 14:30 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:28 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2292.codfw.wmnet
* 14:28 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2291.codfw.wmnet
* 14:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6002.drmrs.wmnet with reason: host reimage
* 14:23 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2291.codfw.wmnet
* 14:23 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2290.codfw.wmnet
* 14:18 zabe@deploy1003: Finished scap sync-world: retry revert (duration: 04m 27s)
* 14:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2290.codfw.wmnet
* 14:17 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2289.codfw.wmnet
* 14:14 bking@cumin1002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 14:14 zabe@deploy1003: Started scap sync-world: retry revert
* 14:12 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2289.codfw.wmnet
* 14:12 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2288.codfw.wmnet
* 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS bookworm
* 14:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2288.codfw.wmnet
* 14:07 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2287.codfw.wmnet
* 14:06 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6002.drmrs.wmnet with reason: reimage
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 14:03 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2287.codfw.wmnet
* 14:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2286.codfw.wmnet
* 14:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 13:53 bking@cumin1002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:53 bking@cumin1002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - [[phab:T397227|T397227]]
* 13:52 zabe@deploy1003: sync-world aborted: [[phab:T397912|T397912]] (duration: 04m 03s)
* 13:48 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2283.codfw.wmnet
* 13:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2282.codfw.wmnet
* 13:40 zabe@deploy1003: Started scap sync-world: [[phab:T397912|T397912]]
* 13:39 _joe_: repooling cp7006, testing logging improvements
* 13:37 vgutierrez: switch upload@eqsin to the new upload cert - [[phab:T394484|T394484]]
* 13:35 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2282.codfw.wmnet
* 13:35 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2281.codfw.wmnet
* 13:30 zabe@deploy1003: zabe: Continuing with sync
* 13:30 moritzm: failover Ganeti master in drmrs02 to ganeti6004 [[phab:T382513|T382513]]
* 13:30 zabe@deploy1003: zabe: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:29 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2281.codfw.wmnet
* 13:29 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2280.codfw.wmnet
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet
* 13:27 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165846{{!}}group1: Set categorylinks to read new (T397912)]]
* 13:27 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet
* 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to drbd
* 13:24 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2280.codfw.wmnet
* 13:24 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2279.codfw.wmnet
* 13:21 _joe_: depooling cp7006 for testing
* 13:18 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2279.codfw.wmnet
* 13:18 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2278.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to drbd
* 13:18 moritzm: installing rsyslog bugfix updates from Bookworm point release
* 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:17 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] (duration: 11m 16s)
* 13:13 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2278.codfw.wmnet
* 13:13 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2277.codfw.wmnet
* 13:13 jgreen@dns1004: END - running authdns-update
* 13:11 jgreen@dns1004: START - running authdns-update
* 13:11 samtar@deploy1003: samtar, eggroll97: Continuing with sync
* 13:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to drbd
* 13:08 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2277.codfw.wmnet
* 13:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2276.codfw.wmnet
* 13:08 samtar@deploy1003: samtar, eggroll97: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 13:05 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1162158{{!}}Assign oathauth-verify-user to default bureaucrat (T265726)]], [[gerrit:1164637{{!}}Add abusefilter-revert to sysops on testwiki (T398107)]]
* 13:02 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2276.codfw.wmnet
* 13:02 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2275.codfw.wmnet
* 12:58 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)
* 12:57 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2275.codfw.wmnet
* 12:57 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2274.codfw.wmnet
* 12:55 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to drbd
* 12:52 urbanecm@deploy1003: urbanecm: Continuing with sync
* 12:52 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2274.codfw.wmnet
* 12:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2273.codfw.wmnet
* 12:51 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165865{{!}}[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866{{!}}[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]
* 12:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2273.codfw.wmnet
* 12:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2272.codfw.wmnet
* 12:41 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2271.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 12:36 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2271.codfw.wmnet
* 12:36 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2270.codfw.wmnet
* 12:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2269.codfw.wmnet
* 12:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2268.codfw.wmnet
* 12:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2267.codfw.wmnet
* 12:14 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2266.codfw.wmnet
* 12:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:09 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2266.codfw.wmnet
* 12:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2265.codfw.wmnet
* 12:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:07 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:06 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2265.codfw.wmnet
* 12:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2264.codfw.wmnet
* 11:58 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2263.codfw.wmnet
* 11:53 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2263.codfw.wmnet
* 11:52 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2262.codfw.wmnet
* 11:47 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2261.codfw.wmnet
* 11:47 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:42 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:42 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2261.codfw.wmnet
* 11:42 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2260.codfw.wmnet
* 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to drbd
* 11:37 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2260.codfw.wmnet
* 11:37 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2259.codfw.wmnet
* 11:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
* 11:33 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
* 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2259.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2258.codfw.wmnet
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:26 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2258.codfw.wmnet
* 11:26 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2257.codfw.wmnet
* 11:21 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2257.codfw.wmnet
* 11:20 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2256.codfw.wmnet
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:16 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to drbd
* 11:15 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2256.codfw.wmnet
* 11:15 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2255.codfw.wmnet
* 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti6004.drmrs.wmnet to cluster drmrs02 and group B13
* 11:10 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2255.codfw.wmnet
* 11:09 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2254.codfw.wmnet
* 11:04 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2253.codfw.wmnet
* 11:00 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2252.codfw.wmnet
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
* 10:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2252.codfw.wmnet
* 10:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2251.codfw.wmnet
* 10:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
* 10:50 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2251.codfw.wmnet
* 10:50 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:49 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2250.codfw.wmnet
* 10:49 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:48 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
* 10:48 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 37271
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:47 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:47 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 137236
* 10:47 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:46 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 137236
* 10:44 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2250.codfw.wmnet
* 10:44 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2249.codfw.wmnet
* 10:44 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:43 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:43 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:42 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti6004.drmrs.wmnet with OS bookworm
* 10:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2249.codfw.wmnet
* 10:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker2248.codfw.wmnet
* 10:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 10:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 10:33 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker2248.codfw.wmnet
* 10:33 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:32 klausman@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 10:28 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 10:27 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:26 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye
* 10:21 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:21 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye
* 10:18 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti6004.drmrs.wmnet with reason: host reimage
* 10:13 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1003&quot;
* 10:08 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] (duration: 09m 52s)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1001.eqiad.wmnet
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:04 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 10:03 kharlan@deploy1003: kharlan: Continuing with sync
* 10:02 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 10:01 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:00 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:58 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165836{{!}}UserInfoCard: prevent default link behavior with &quot;click&quot; (T398323)]]
* 09:57 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 09:55 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup1001.eqiad.wmnet
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti6004.drmrs.wmnet with OS bookworm
* 09:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:53 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti6004.drmrs.wmnet with reason: reimage
* 09:50 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh6002.wikimedia.org to plain
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of doh6002.wikimedia.org to plain
* 09:49 vgutierrez: acme-chief: stop issuing RSA certificates by default - [[phab:T398020|T398020]]
* 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum6002.drmrs.wmnet to plain
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2001.codfw.wmnet
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:47 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:47 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002&quot;
* 09:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum6002.drmrs.wmnet to plain
* 09:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:45 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus6002.drmrs.wmnet to plain
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:44 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes: api auth and bwlimit rules - oblivian@cumin1003
* 09:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Bugfixes: api auth and bwlimit rules - oblivian@cumin1003&quot;
* 09:42 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir6002.drmrs.wmnet to plain
* 09:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:39 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow6001.drmrs.wmnet to plain
* 09:37 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts backup2001.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet
* 09:36 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] (duration: 10m 15s)
* 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet
* 09:30 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 09:29 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 09:28 zabe@deploy1003: zabe: Continuing with sync
* 09:27 zabe@deploy1003: zabe: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:25 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165831{{!}}Reapply &quot;categorylinks: Set group0 to read new&quot; (T397912)]]
* 09:23 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] (duration: 08m 26s)
* 09:18 zabe@deploy1003: zabe: Continuing with sync
* 09:17 zabe@deploy1003: zabe: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165827{{!}}Fix categorylinks join order and use index on correct table (T398380)]]
* 09:06 volans: uploaded debmonitor-server,python3-debmonitor_0.6.4 to apt.wikimedia.org bookworm-wikimedia
* 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet
* 09:06 jmm@dns1004: END - running authdns-update
* 09:05 jmm@dns1004: START - running authdns-update
* 09:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet
* 09:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti3006.esams.wmnet to cluster esams02 and group BW27
* 09:01 moritzm: rebalance ganeti/eqsin following Bookworm reimages
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5007.eqsin.wmnet to cluster eqsin and group 1
* 08:53 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet
* 08:34 jmm@dns1004: END - running authdns-update
* 08:33 jmm@dns1004: START - running authdns-update
* 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5007.eqsin.wmnet with OS bookworm
* 08:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 08:20 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 08:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5007.eqsin.wmnet with reason: host reimage
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 07:50 jmm@dns1004: END - running authdns-update
* 07:49 jmm@dns1004: START - running authdns-update
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5007.eqsin.wmnet with OS bookworm
* 07:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5007.eqsin.wmnet with reason: reimage
* 06:29 Amir1: dropping l10n_cache table everywhere ([[phab:T397367|T397367]])
* 06:28 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 06:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78735 and previous config saved to /var/cache/conftool/dbconfig/20250702-061517-ladsgroup.json
* 06:04 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 06:02 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:58 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 05:57 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - [[phab:T397300|T397300]]
* 02:50 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 02:32 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:28 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 02:12 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bullseye
* 00:53 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm

== 2025-07-01 ==
* 23:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage
* 23:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:22 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage
* 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
* 23:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
* 23:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye
* 23:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] (duration: 08m 49s)
* 22:58 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Continuing with sync
* 22:57 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye
* 22:57 zabe@deploy1003: zabe: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye
* 22:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165615{{!}}Revert &quot;categorylinks: Set group0 to read new&quot; (T397912 T398380)]]
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:54 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: sync - dzahn@cumin1002&quot;
* 22:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:53 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] (duration: 08m 40s)
* 22:49 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:47 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:47 zabe@deploy1003: zabe: Continuing with sync
* 22:46 zabe@deploy1003: zabe: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts miscweb1003.eqiad.wmnet
* 22:45 dzahn@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 22:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1165540{{!}}categorylinks: Set group0 to read new (T397912)]]
* 22:44 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:44 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:36 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:35 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] (duration: 29m 56s)
* 22:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:34 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:31 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb1003.eqiad.wmnet
* 22:30 toyofuku@deploy1003: bwang, toyofuku: Continuing with sync
* 22:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb2003.codfw.wmnet
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:28 dzahn@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:28 dzahn@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: miscweb2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002&quot;
* 22:26 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:23 dzahn@cumin1002: START - Cookbook sre.dns.netbox
* 22:22 ejegg: payments-wiki upgraded from {{Gerrit|a92f03c3}} to {{Gerrit|9c7f3a73}}
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1092.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1093.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:18 dzahn@cumin1002: START - Cookbook sre.hosts.decommission for hosts miscweb2003.codfw.wmnet
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:17 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update dns ms-be1092,934 - jclark@cumin1002&quot;
* 22:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:10 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:09 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:07 toyofuku@deploy1003: bwang, toyofuku: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:06 ejegg: fundraising scheduled jobs restarted
* 22:05 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165585{{!}}Update mobile search overlay temporary input styles]]
* 22:04 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 22:02 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb2003.codfw.wmnet with reason: decom
* 22:01 dzahn@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on miscweb1003.eqiad.wmnet with reason: decom
* 21:59 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] (duration: 10m 10s)
* 21:59 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
* 21:56 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
* 21:54 toyofuku@deploy1003: toyofuku, bwang: Continuing with sync
* 21:51 toyofuku@deploy1003: toyofuku, bwang: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:51 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:51 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:49 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1165549{{!}}Enable mobile search recommendations in all eligible wikis except enwiki]]
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 21:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 21:25 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 21:06 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 21:03 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 20:48 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:45 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:45 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 20:41 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] (duration: 10m 35s)
* 20:39 ejegg: fundraising civicrm upgraded from {{Gerrit|5ae93148}} to {{Gerrit|521d0dbe}}
* 20:36 cjming@deploy1003: zhaofjx, cjming: Continuing with sync
* 20:33 cjming@deploy1003: zhaofjx, cjming: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:31 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1163483{{!}}zhwiki: Permissions change for abusefilter groups (T397788)]]
* 20:26 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:26 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:24 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
* 20:20 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 20:04 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:04 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye
* 20:03 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 20:02 ejegg: disabled queue consumers for segment updates
* 19:50 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:50 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:47 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye
* 19:43 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:42 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:37 jhathaway@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
* 19:26 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s)
* 19:23 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:20 kemayo@deploy1003: kemayo: Continuing with sync
* 19:20 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage
* 19:19 kemayo@deploy1003: kemayo: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1165589{{!}}Edit check: fix counter logging for SLO (T395444)]]
* 19:00 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm
* 19:00 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: [[phab:T383173|T383173]]
* 17:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 andrew@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:55 andrew@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003&quot;
* 16:51 andrew@cumin1003: START - Cookbook sre.dns.netbox
* 16:45 andrew@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet
* 16:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json
* 16:37 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 16:37 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 16:13 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s)
* 16:11 inflatador: bking@prometheus1005:~$ sudo run-puppet-agent [[phab:T398341|T398341]]
* 16:10 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:08 jhancock@cumin1003: START - Cookbook sre.dns.netbox
* 16:07 swfrench@deploy1003: swfrench: Continuing with sync
* 16:06 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013
* 16:06 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc2013
* 16:06 swfrench@deploy1003: swfrench: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:04 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1152295{{!}}Remove title-case overrides for PHP 8.1 migration (T394556)]]
* 16:01 swfrench-wmf: finished page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:54 swfrench-wmf: starting page renames for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:51 swfrench-wmf: renamed 1 user for Unicode title-case transition - [[phab:T396903|T396903]]
* 15:44 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs7003.magru.wmnet
* 15:44 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs7003.magru.wmnet
* 15:37 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 15:37 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 15:35 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs7003.magru.wmnet with reason: katran migration
* 15:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:25 ejegg: SmashPig upgraded from {{Gerrit|8486f9fb}} to {{Gerrit|52397453}}
* 15:21 ejegg: SmashPig upgraded from {{Gerrit|bdc59e01}} to {{Gerrit|8486f9fb}}
* 15:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet
* 15:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 15:10 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]] (duration: 00m 37s)
* 15:09 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab1004 for [[phab:T398328|T398328]]
* 15:09 brennen@deploy1003: Finished deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]] (duration: 00m 41s)
* 15:08 brennen@deploy1003: Started deploy [phabricator/deployment@311587a]: deploy phab2002 for [[phab:T398328|T398328]]
* 15:08 ejegg: standalone SmashPig upgraded from {{Gerrit|ad4baa32}} to {{Gerrit|bdc59e01}}
* 15:08 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 15:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 15:04 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 15:01 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 15:00 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:55 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:54 moritzm: failover Ganeti master in eqsin to ganeti5004
* 14:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet
* 14:46 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin and group 1
* 14:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet
* 14:26 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 14:25 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 13:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm
* 13:51 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 13:51 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] (duration: 09m 44s)
* 13:45 zabe@deploy1003: zabe: Continuing with sync
* 13:44 zabe@deploy1003: zabe: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:41 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1164472{{!}}categorylinks: Set testwiki to read new (T397912)]]
* 13:40 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:39 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:37 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:36 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:35 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:29 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:23 urbanecm@deploy1003: urbanecm, cyndywikime: Continuing with sync
* 13:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage
* 13:13 urbanecm@deploy1003: urbanecm, cyndywikime: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 jmm@dns1004: END - running authdns-update
* 13:11 jmm@dns1004: START - running authdns-update
* 13:10 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1164979{{!}}Growth: Configure higher impact module edit limits for english and test wiki (T341599)]]
* 12:59 XioNoX: setup BGP to Paylb on pfw1-eqiad - [[phab:T397865|T397865]]
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm
* 12:57 jmm@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5006.eqsin.wmnet with reason: reimage
* 12:53 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:49 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 12:48 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 12:45 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1004.eqiad.wmnet
* 12:39 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1003.eqiad.wmnet
* 12:38 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:38 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:38 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:35 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 12:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 12:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 12:32 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:32 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:31 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1003.eqiad.wmnet
* 12:31 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 12:31 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 12:29 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1002.eqiad.wmnet
* 12:23 jmm@dns1004: END - running authdns-update
* 12:22 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:22 jmm@dns1004: START - running authdns-update
* 12:21 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1002.eqiad.wmnet
* 12:21 moritzm: installing libcap2 security updates
* 12:20 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl1001.eqiad.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 12:15 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet
* 12:13 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl1001.eqiad.wmnet
* 12:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet
* 12:07 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2005.codfw.wmnet
* 12:02 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2005.codfw.wmnet
* 12:00 moritzm: manually clean out external_cloud_vendors directory on puppet 5 frontends to fix Puppet runs
* 11:59 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2004.codfw.wmnet
* 11:54 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2004.codfw.wmnet
* 11:53 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet
* 11:47 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 11:47 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 11:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 11:45 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2003.codfw.wmnet
* 11:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 11:43 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2002.codfw.wmnet
* 11:37 jmm@dns1004: END - running authdns-update
* 11:36 jmm@dns1004: START - running authdns-update
* 11:35 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2002.codfw.wmnet
* 11:30 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 11:30 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 11:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet
* 11:01 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2004.codfw.wmnet
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet
* 10:54 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp2004.codfw.wmnet
* 10:50 root@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-ctrl2001.codfw.wmnet
* 10:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet
* 10:33 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2007.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 10:32 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet
* 10:27 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:26 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2050.codfw.wmnet to cluster codfw and group B
* 10:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2004.wikimedia.org
* 10:19 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 10:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: idp-test2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 10:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 10:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 10:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:08 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Switch to 10G ([[phab:T378715|T378715]])
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T378715|T378715]]', diff saved to https://phabricator.wikimedia.org/P78729 and previous config saved to /var/cache/conftool/dbconfig/20250701-100729-ladsgroup.json
* 10:06 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts idp-test2004.wikimedia.org
* 09:59 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2007.codfw.wmnet with reason: Maintenance and reboot
* 09:57 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2006.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet
* 09:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin and group 1
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet
* 09:33 hashar@deploy1003: Finished deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]] (duration: 00m 12s)
* 09:32 hashar@deploy1003: Started deploy [gerrit/gerrit@4e671a0]: Remove all references to patchdemo legacy - [[phab:T391866|T391866]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet
* 09:25 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: sync
* 09:25 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: sync
* 09:21 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2006.codfw.wmnet with reason: Maintenance and reboot
* 09:17 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2005.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 09:11 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] (duration: 09m 15s)
* 09:05 kharlan@deploy1003: kharlan: Continuing with sync
* 09:04 kharlan@deploy1003: kharlan: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:02 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1165470{{!}}UserInfoCard: Fix opt-in to temporary account label display (T395661)]], [[gerrit:1165265{{!}}UserInfoCard can unintentionally render information for more than one user]]
* 09:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:55 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:55 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:54 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:53 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:44 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 08:44 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: Maintenance and reboot
* 08:42 jynus@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup2004.codfw.wmnet: Renew puppet certificate - jynus@cumin1002
* 08:38 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:34 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 08:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage
* 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 08:08 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm
* 08:08 moritzm: installing sudo security updates
* 08:07 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2050.codfw.wmnet with OS bookworm
* 07:58 urbanecm: Manually start a Growth cron job via `kubectl create job growthexperiments-deleteoldsurveys-$(date +&quot;%Y%m%d%H%M&quot;) --from=cronjobs/growthexperiments-deleteoldsurveys` to verify whether a recent failure is permanent
* 07:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Corvus out of all services on: 2396 hosts
* 07:54 vgutierrez: switching upload@ulsfo to upload TLS certificate - [[phab:T394484|T394484]]
* 07:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:48 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2050.codfw.wmnet with reason: host reimage
* 07:43 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] (duration: 12m 04s)
* 07:43 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.ulsfo.wmnet
* 07:43 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2004.codfw.wmnet with reason: Maintenance and reboot
* 07:38 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.ulsfo.wmnet
* 07:38 urbanecm@deploy1003: urbanecm, daniuu: Continuing with sync
* 07:37 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5005.eqsin.wmnet with reason: reimage
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti2050.codfw.wmnet with OS bookworm
* 07:33 urbanecm@deploy1003: urbanecm, daniuu: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:31 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1165056{{!}}nlwiki: add VRT agent user group (T398216)]]
* 07:16 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] (duration: 14m 17s)
* 07:09 kartik@deploy1003: kartik: Continuing with sync
* 07:06 kartik@deploy1003: kartik: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:02 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1164948{{!}}Remove cxstats campaign (T393705)]]
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.5 (duration: 01m 38s)
* 03:58 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]] (duration: 55m 48s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.8  refs [[phab:T392178|T392178]]
* 02:13 ejegg: payments-wiki upgraded from {{Gerrit|52f6940f}} to {{Gerrit|a92f03c3}}
* 01:46 ejegg: fundraising civicrm upgraded from {{Gerrit|e35d3778}} to {{Gerrit|5ae93148}}
* 00:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 00:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 00:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 00:01 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply

== Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>fp2d1shwzkhxi2r2cb04hm6djvatnxm</sha1>
    </revision>
  </page>
  <page>
    <title>Release Engineering/SAL</title>
    <ns>0</ns>
    <id>17290</id>
    <revision>
      <id>2320897</id>
      <parentid>2320845</parentid>
      <timestamp>2025-07-07T08:45:26Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>hashar: gerrit: change mediawiki/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; | T390719</comment>
      <origin>2320897</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="56075" sha1="euw8qzbeumyrymae1r69gd8z7yz1pgf" xml:space="preserve">== 2025-07-07 ==
* 08:45 hashar: gerrit: change mediawiki/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]

== 2025-07-04 ==
* 21:45 Krinkle: [[phab:T289318|T289318]]: Change stream_config_uri in Hiera (Horizon instance config for  deployment-eventgate-4 and deployment-eventstreams-2 ) from https://meta.wikimedia.beta.wmflabs.org/w/api.php?action=streamconfigs to https://meta.wikimedia.beta.wmcloud.org/w/api.php?action=streamconfigs
* 21:45 Krinkle: [[phab:T289318|T289318]]: Change profile::cache::varnish::frontend::fe_vcl_config/static_host in Hiera (Horizon puppet prefix for cache-text and cache-upload) from  en.wikipedia.beta.wmflabs.org to en.wikipedia.beta.wmcloud.org
* 21:41 Krinkle: Change profile::docker::runner::service_defs/mediawiki-services-cxserver/mwapi_req/host in  Horizon (Hiera puppet prefix) from en.wikipedia.beta.wmflabs.org to en.wikipedia.beta.wmcloud.org. [[phab:T289318|T289318]]
* 21:39 Krinkle: Change profile::docker::runner::service_defs/mediawiki-services-push-notifications/mwapi_req/host in  Horizon (Hiera puppet prefix) from meta.wikimedia.beta.wmflabs.org to meta.wikimedia.beta.wmcloud.org. [[phab:T289318|T289318]]
* 13:49 hashar: gerrit: deleted project glam/gwtoolset {{!}} Created October 11st 2012 and has never been used
* 13:24 hashar: gerrit: changed `All-Projects` default submit strategy to `Rebase if Necessary`. Does not affect mediawiki/* or operations/* among others # [[phab:T390719|T390719]]

== 2025-07-02 ==
* 21:41 Krinkle: [[phab:T289318|T289318]] - Change service::catalog probes for mw-api-int in Horizon prefix Puppet from en.wikipedia.beta.wmflabs.org/w/api.php to en.wikipedia.beta.wmcloud.org/w/api.php
* 21:38 Krinkle: [[phab:T289318|T289318]] - Change profile::mail::mx::verp_bounce_post_url in Horizon prefix puppet, from https://meta.wikimedia.beta.wmflabs.org/w/api.php to https://meta.wikimedia.beta.wmcloud.org/w/api.php.
* 17:33 hashar: Reloaded Zuul for &quot;Drop generic ruby rake jobs&quot; https://gerrit.wikimedia.org/r/c/integration/config/+/1165947/
* 14:51 hashar: Zuul: Upgrade translatewiki-ruby* from 2.5 to 2.7, for [[phab:T335765|T335765]]
* 14:13 James_F: Zuul: Upgrade ooui-ruby* from 2.5 to 2.7, for [[phab:T335765|T335765]]
* 07:47 hashar: gerrit: ssh -p 29418 gerrit.wikimedia.org rename-project operations/debs/wmf-sre-laptop operations/debs/wmf-laptop # [[phab:T365985|T365985]]

== 2025-07-01 ==
* 10:32 hashar: gerrit: deleted secrets/wikimetrics , a 2016 experiment to hold credentials for deployment purpose # [[phab:T219334|T219334]]
* 08:21 hashar: gerrit: archived https://gerrit.wikimedia.org/g/qrpedia Latest source code is elsewhere {{!}} [[phab:T244135|T244135]]
* 07:41 hashar: Disabled CI for REL1_42 # [[phab:T389313|T389313]]

== 2025-06-30 ==
* 22:09 bd808: Blocked 4 Class C networks with &gt;1000 hits in the last 100,000 Beta Cluster requests
* 21:40 bd808: Unblocked 46.28.80.0/21 at CDN edge ([[phab:T398124|T398124]])
* 20:17 bd808: Upgraded haproxy to 2.8.14-1~bpo11+1 on deployment-cache-text08 ([[phab:T398176|T398176]])
* 20:13 bd808: Upgraded haproxy to 2.8.14-1~bpo11+1 on deployment-cache-upload08 ([[phab:T398176|T398176]])
* 20:03 bd808: Remove `profile::cache::haproxy::version: haproxy26` from deployment-cache Prefix Puppet ([[phab:T398176|T398176]])
* 17:31 hashar: gerrit: marked read-only all operations/debs/contenttranslation/apertium* repositories. Untouched since 2020.
* 16:37 hashar: gerrit: change wikimedia/fundraising/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:57 hashar: gerrit: change labs/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:37 hashar: gerrit: change mediawiki/libs/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:31 hashar: gerrit: change performance/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:28 hashar: gerrit: deleted videojs-resolution-switcher and videojs-responsive-layout , forks of other projects with no local modifications/changes.

== 2025-06-27 ==
* 14:12 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1164451

== 2025-06-26 ==
* 14:49 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1164197 ([[phab:T397922|T397922]])
* 14:43 dancy: Updated gitlab-cloud-runners to gitlab-runner v17.11.3 ([[phab:T397899|T397899]])
* 10:55 urbanecm: deployment-prep: Run `foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/importOresTopics.php --count=20000 --verbose` ([[phab:T393684|T393684]])

== 2025-06-25 ==
* 21:16 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1163883/1 to deployment-puppetserver-1 ([[phab:T397877|T397877]])
* 20:24 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/3 to deployment-puppetserver-1 ([[phab:T397872|T397872]])
* 18:19 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/2 to deployment-puppetserver-1 ([[phab:T397717|T397717]])
* 17:05 thcipriani: Upgrading scap to 4.182.0 in beta cluster
* 08:55 hashar: jenkins: updated job publish-to-doc to use label productionAgents rather than contint1002 # [[phab:T397815|T397815]]
* 08:52 hashar: jenkins: updated jobs fail-archived-repositories, train-deploy-notes and trigger-* to use label productionAgents rather than contint1002 # [[phab:T397815|T397815]]
* 02:19 Krinkle: Add mapping for performance.wikimedia.beta.wmcloud.org to profile::trafficserver::backend::mapping_rules in Hiera under deployment-cache-text prefix. Same mapping as the wmflabs version. [[phab:T289318|T289318]]

== 2025-06-23 ==
* 16:41 greg-g: removed 2fa from XenoRyet, confirmed on video call
* 16:05 dancy: Ran `docker run --rm -it --network gitlab-runner --entrypoint buildctl docker-registry.wikimedia.org/repos/releng/buildkit:wmf-v0.22.0 --addr buildkitd:1234 prune` on `runner-1025.gitlab-runners.eqiad1.wikimedia.cloud
* 07:20 James_F: Zuul: [mediawiki/extensions/EventLogging] Add CodeEditor Phan dependency, for [[phab:T346540|T346540]]

== 2025-06-22 ==
* 21:42 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1162179

== 2025-06-21 ==
* 02:54 Krinkle: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/1162106

== 2025-06-20 ==
* 18:57 dduvall: ran `helm --namespace gitlab-runner uninstall docker-hub-mirror` to fix helm state. reapplying production cluster configuration
* 18:41 dduvall: deleted docker-hub-mirror statefulset and admission controller deployment. reapplying production cluster configuration
* 18:18 dduvall: seeing numerous image pull errors in gitlab-cloud-runner cluster

== 2025-06-19 ==
* 09:38 sergi0: deployment-prep: GrowthExperiments config migration `foreachwiki extensions/CommunityConfiguration/maintenance/migrateConfig.php GrowthSuggestedEdits` — [[phab:T393771|T393771]]
* 09:18 urbanecm: deployment-prep: Update changeprop config perhttps://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1161443 using [[wikitech:Changeprop#To_deployment-prep]] ([[phab:T394958|T394958]]; this time actually changing the beta config)
* 09:10 urbanecm: deployment-prep: Update changeprop config per https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1150699 using [[wikitech:Changeprop#To_deployment-prep]] ([[phab:T394958|T394958]])

== 2025-06-18 ==
* 23:26 bd808: Blocked 128.241.0.0/16 &quot;NTT America&quot; network. ([[phab:T397378|T397378]])
* 22:10 bd808: Blocked 202.76.160.0/20 &quot;Huawei-Cloud-SG&quot; network. ([[phab:T397378|T397378]])
* 22:02 bd808: Blocked 146.174.160.0/19 &quot;Huawei-Cloud-SG&quot; network. ([[phab:T397378|T397378]])
* 18:19 bd808: `docker system prune --all` on runner-1023.gitlab-runners.eqiad1.wikimedia.cloud
* 13:10 James_F: Zuul: Add EggRoll97 to CI allowlist
* 13:08 James_F: Zuul: Add James E. Blair to CI allowlist
* 13:06 James_F: Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template
* 04:14 Krinkle: Fix profile::trafficserver::backend::mapping_rules in deployment-cache-text to include `rb-mw-mangling-beta.lua` as otherwise w.beta.wmcloud.org serves 404 Domain Not Configured, ref [[phab:T289318|T289318]], [[phab:T396012|T396012]]
* 04:13 Krinkle: Fix profile::trafficserver::backend::mapping_rules in deployment-cache-upload to include `rb-mw-mangling-beta.lua` as otherwise w.beta.wmcloud.org serves 404 Domain Not Configured, ref [[phab:T289318|T289318]], [[phab:T396012|T396012]]
* 04:10 Krinkle: Change shortener_domain in deployment-cache-text prefix from `w-beta.wmflabs.org` to `w.beta.wmcloud.org`, to apply VCL normalization for w.wiki in Beta, ref [[phab:T289318|T289318]], [[phab:T396012|T396012]]

== 2025-06-16 ==
* 15:15 James_F: Docker: [quibble-bullseye] Add the MariaDB binaries to our path [[phab:T366646|T366646]]
* 14:32 James_F: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again, for [[phab:T366646|T366646]]

== 2025-06-13 ==
* 15:50 James_F: Docker: Drop php-ast image, now unused, for [[phab:T396312|T396312]]
* 15:48 James_F: Zuul: Drop broken composer-coverage-patch job from the two repos using it

== 2025-06-12 ==
* 20:41 bd808: `sudo service varnish-frontend restart` on deployment-cache-text08 to pick up blocked_nets changes ([[phab:T394881|T394881]])
* 20:28 bd808: `sudo service varnish-frontend restart` on deployment-cache-text08 to pick up blocked_nets changes ([[phab:T396748|T396748]])
* 20:15 bd808: Added `profile::memcached::firewall_srange: ~` to deployment-memc Puppet prefix ([[phab:T396732|T396732]])
* 16:24 James_F: Docker: Cascade uses of php* with new php-ast inline build, for [[phab:T396312|T396312]]
* 15:23 dancy: Upgraded gitlab-cloud-runners to v17.10.2 ([[phab:T396701|T396701]])
* 15:04 James_F: Docker: [node-test-brower-php*-composer] Build php-ast inline, for [[phab:T396312|T396312]]
* 14:50 James_F: Docker: [php*] Build php-ast with the exact same PHP version, for [[phab:T396312|T396312]]

== 2025-06-10 ==
* 22:53 James_F: Zuul: [css-sanitizer] Add coverage reporting
* 20:02 brennen: Updating buildkitd to v0.22.0 in gitlab-cloud-runners ([[phab:T394931|T394931]])
* 14:37 James_F: Zuul: [maps/*] Mark all as archived
* 13:33 sergi0: run migration in GrowthSuggestedEditsSchema `foreachwikiindblist growthexperiments extensions/CommunityConfiguration/maintenance/migrateConfig.php GrowthSuggestedEdits` [[phab:T395383|T395383]]
* 13:31 sergi0: set version in GrowthSuggestedEdits  schema `foreachwiki extensions/CommunityConfiguration/maintenance/setVersionData.php GrowthSuggestedEdits 1.0.0`
* 11:35 James_F: jforrester@integration-castor05:/srv/castor$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-rundoc/ # [[phab:T396426|T396426]]

== 2025-06-09 ==
* 15:01 James_F: Zuul: [labs/tools/WdTmCollab] Add tox job CI, for [[phab:T396349|T396349]]
* 14:25 James_F: Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Mark as archived, for [[phab:T396311|T396311]]
* 14:16 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Test on PHP 8.4, for [[phab:T386570|T386570]]

== 2025-06-08 ==
* 18:14 James_F: Zuul: [mediawiki/extensions/Echo] Remove EventLogging
* 18:12 James_F: Zuul: Fold extension-quibble-php81-or-later template into extension-quibble
* 18:04 James_F: Zuul: [mediawiki/extensions/SemanticVersion] Add basic CI

== 2025-06-06 ==
* 14:37 jnuche: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/79

== 2025-06-05 ==
* 23:21 thcipriani: update scap in beta to 4.171.0 to match prod
* 20:44 James_F: Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base, archive repo's CI, for [[phab:T354310|T354310]]
* 20:20 bd808: Added `profile::memcached::firewall_src_sets: ~` to deployment-memc prefix puppet ([[phab:T396109|T396109]])
* 19:03 Krinkle: Update profile::tlsproxy::envoy::cfssl_options under deployment-mediawiki in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary. ref [[phab:T289318|T289318]]
* 18:26 James_F: Docker: Re-build PHP images with php-uuid (and incidentally bump versions), for [[phab:T373752|T373752]]
* 17:14 James_F: Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81
* 17:10 James_F: Docker: [phpmetrics] Migrate parent image from php74 to php81
* 17:10 James_F: Where will Abstract Content go?
* 17:07 James_F: Zuul: [mediawiki/extensions/WikimediaMaintenance] Add dependencies, for [[phab:T58074|T58074]]
* 16:39 James_F: Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Use a template for CI
* 16:37 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Stop testing in PHP 7.4
* 16:36 James_F: Zuul: [labs/tools/heritage] Raise PHP testing from 7.4 to 8.1
* 16:34 James_F: Zuul: Stop testing most libraries and tools in PHP 7.4
* 16:28 James_F: Zuul: Stop testing PHP extensions with PHP 7.4
* 16:26 James_F: Zuul: [integration/quibble] Stop testing in PHP 7.4, for [[phab:T328921|T328921]] and [[phab:T328922|T328922]]
* 16:23 James_F: Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4
* 16:21 James_F: Zuul: [operations/mediawiki-config] Stop testing in PHP 7.4
* 16:09 James_F: Zuul: Drop all PHP 7.4 testing for MediaWiki things, for [[phab:T328921|T328921]] and [[phab:T328922|T328922]]
* 04:46 Krinkle: gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$  Cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1153764, ref [[phab:T289318|T289318]]
* 03:58 Krinkle: Update profile::cache::haproxy::available_unified_certificates under deployment-cache in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary. Remove `*.zero.wikipedia.beta.wmflabs.org` which wasn't responding/didn't work anymore. ref [[phab:T289318|T289318]]
* 03:34 Krinkle: Update profile::acme_chief::certificates under deployment-acme-chief prefix in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary (wikipedia and wikivoyage were already there), ref [[phab:T289318|T289318]]
* 03:34 Krinkle: Update profile::acme_chief::certificates under deployment-acme-chief prefix in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary (wikipedia and wikivoyage were already there)
* 00:32 Krinkle: Add `TXT *.wikimedia.beta.wmcloud.org. &quot;v=spf1 -all&quot;` to match beta.wmflabs.org DNS (ref [[phab:T289318|T289318]], changing email is out of scope for now, but might as well add the DNS records).
* 00:22 Krinkle: Adding missing DNS entries under beta.wmcloud.org. There was already: *.wikipedia, *.m.wikimedia, *.wikivoyage, *.m.wikivoyage (for [[phab:T355281|T355281]]). Adding: wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary, wikidata, upload ([[phab:T289318|T289318]]).

== 2025-06-04 ==
* 21:27 James_F: Zuul: [mediawiki/extensions/Springboard] Add basic CI, for [[phab:T395981|T395981]]
* 12:10 lucaswerkmeister: lucaswerkmeister@deployment-deploy04:~$ mwscript createAndPromote commonswiki --interface-admin --force 'Lucas Werkmeister' # w-beta.wmflabs.org/mt

== 2025-06-03 ==
* 23:59 James_F: Zuul: [mediawiki/services/&lt;some&gt;] Upgrade test suite to Node 24 &amp; 22, for [[phab:T395926|T395926]]
* 23:56 James_F: Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22, for [[phab:T395926|T395926]]
* 23:56 James_F: Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22, for [[phab:T395926|T395926]]
* 23:55 James_F: Zuul: [oojs/*i] Upgrade test suite to Node 24 and Node 22, for [[phab:T395926|T395926]]
* 23:53 James_F: Zuul: [wikimedia/portals/deploy] Drop tests, this repo isn't testable
* 23:20 James_F: Zuul: Provide experimental Node 24 jobs where Node 22 ones exist, for [[phab:T395926|T395926]]
* 17:09 bd808: Forced puppet run on deployment-webperf21 to pick up Kafka config changes ([[phab:T391273|T391273]])
* 17:08 bd808: Manually expanded (duplicated) jumbo-eqiad and main-eqiad aliases in kafka_clusters hiera config ([[phab:T391273|T391273]])
* 17:04 bd808: Added jumbo-eqiad and main-eqiad aliases to kafka_clusters hiera config ([[phab:T391273|T391273]])
* 16:00 James_F: Docker: Provide initial Node 24 images, for [[phab:T395923|T395923]]
* 09:53 TheresNoTime: `samtar@deployment-cache-text08:~$ sudo service varnish-frontend restart` for [[phab:T395808|T395808]]
* 09:52 TheresNoTime: `samtar@deployment-cache-text08:~$ sudo -i puppet agent -tv` for [[phab:T395808|T395808]]

== 2025-06-02 ==
* 14:37 James_F: Zuul: Add Matrix to CI allowlist
* 14:37 James_F: Zuul: [operations/software/gerrit/plugins/events-wikimedia] mark as archived, for [[phab:T304947|T304947]]
* 14:36 James_F: Zuul: [mediawiki/extensions/CookieConsent] Add basic CI
* 13:45 hashar: Updating  Jenkins jobs for &quot;drop obsolete creation of log &amp; src dirs&quot; {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1152702

== 2025-05-30 ==
* 22:16 thcipriani: killed 1000s of zuul merger jobs via https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions for parsoid, wikibase, and core
* 21:20 bd808: Poked hole in blocked_nets for 188.214.8.0/21 ([[phab:T395709|T395709]])
* 09:43 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 57273 and 57274

== 2025-05-29 ==
* 22:18 bd808: Submitted WikimediaDebug v3.1.0 to addons.mozilla.org for review ([[phab:T395190|T395190]], [[phab:T315111|T315111]])
* 22:12 bd808: Submitted WikimediaDebug v3.1.0 to Chrome Web Store for review ([[phab:T395190|T395190]], [[phab:T315111|T315111]])

== 2025-05-28 ==
* 20:27 James_F: Zuul: [mediawiki/extensions/ArticleSummaries] Promote to Wikimedia production, for [[phab:T393940|T393940]]
* 13:15 James_F: [Beta Cluster] On deployment-deploy04, running DELETE FROM localuser WHERE lu_wiki='en_rtlwiki'; and DELETE FROM localnames WHERE ln_wiki='en_rtlwiki'; as part of closing the wiki
* 12:30 James_F: Zuul: Add an explanatory note to bluespice template that we skip non-LTSes

== 2025-05-24 ==
* 21:52 Krinkle: Disable publishing notifs on Phab tasks from extension-Chart mirror, [[phab:T143162|T143162]], [[phab:T272803|T272803]]

== 2025-05-23 ==
* 18:36 James_F: Zuul: [mediawiki/core] Restore node testing for release branches, for [[phab:T395141|T395141]]
* 17:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1149705

== 2025-05-22 ==
* 21:15 bd808: Forced Puppet run and restarted varnins-frontend on deployment-cache-upload08 to pick up new config ([[phab:T393404|T393404]])
* 21:12 bd808: Forced Puppet run and restarted varnins-frontend on deployment-cache-text08 to pick up new config ([[phab:T393404|T393404]])
* 21:09 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143602 ([[phab:T393404|T393404]])
* 21:09 bd808: Added `block_help: &quot;see https://wikitech.wikimedia.org/wiki/Beta/Blocked_help for more information.&quot;` under `profile::cache::varnish::frontend::fe_vcl_config` in both deployment-cache-text and deployment-cache-upload Prefix Puppet ([[phab:T393404|T393404]])
* 20:11 brennen: devtools: phorge: test deploying work/merge-phorge-2024.35 changes
* 17:25 bd808: `./jjb-update 'selenium-daily-beta*-MediaWiki'` to deploy updates to selenium-daily-beta-MediaWiki and selenium-daily-betacommons-MediaWiki failure notifications ([[phab:T394551|T394551]])
* 14:45 dancy: Upgrade gitlab-runner to v17.10.1 in gitlab-cloud-runner (staging and production) [[phab:T394953|T394953]]
* 11:39 hashar: Triggered replication of mediawiki/extensions/BlueSpiceSmartlist and mediawiki/extensions/BlueSpiceSmartList to fix https://github.com/wikimedia/mediawiki-extensions-BlueSpiceSmartlist {{!}} [[phab:T394903|T394903]]
* 11:37 hashar: gerrit: changed parent of mediawiki/extensions/BlueSpiceSmartlist (lower case L) to All-Archived-Projects to prevent it from being replicated to GitHub {{!}} [[phab:T394903|T394903]]

== 2025-05-21 ==
* 07:24 hashar: restarted Gerrit on gerrit1003
* 07:18 hashar: restarted Jenkins on contint1002

== 2025-05-20 ==
* 17:51 bd808: Open CDN edge blocks to allow traffic from 190.217.20.32/28
* 17:13 dancy: Restarting Jenkins on contint1002
* 16:27 James_F: Docker: [quibble-bullseye-php81-coverage]: Fix clover-edit for py39
* 14:30 James_F: Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15
* 14:28 hashar: integration: cleared Docker build cache on integration-agent-docker-1052 and integration-agent-docker-1061
* 13:49 James_F: Docker: Provide quibble-bullseye-php81-coverage

== 2025-05-19 ==
* 15:48 James_F: Zuul: Switch primary master branch testing to PHP 8.1, not 7.4
* 15:45 James_F: Zuul: Switch / remove any experimental testing to PHP 8.1, not 7.4
* 15:39 James_F: Zuul: Switch REL1_39 branch testing to PHP 8.1, not 7.4
* 15:37 James_F: Zuul: Switch all wmf branch testing to PHP 8.1, not 7.4
* 13:25 James_F: Zuul: Simplify the regular Quibble job name to drop 'noselenium'
* 13:24 James_F: jjb: Simplify the regular Quibble job name to drop 'noselenium'
* 12:18 hashar: integration: cleaned Docker build cache on integration-agent-docker-1045
* 09:26 hashar: integration: cleaned Docker build cache on integration-agent-docker-1040

== 2025-05-16 ==
* 16:57 James_F: Zuul: Split Quibble jobs into selenium-only and non-selenium for skins

== 2025-05-15 ==
* 21:22 bd808: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1146722
* 13:54 James_F: Zuul: [mediawiki/extensions/Realnames] Use vendor quibble, not composer
* 09:34 codders: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1146520

== 2025-05-14 ==
* 21:31 bd808: Restarted varnish-frontend on deployment-cache-text08 to pick up blocked_nets changes ([[phab:T394311|T394311]])
* 16:06 hashar: Updating jobs for &quot;jjb: silence some shell blocks in macro-docker.yaml&quot; {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1145090 {{!}} [[phab:T393847|T393847]]
* 13:43 hashar: Reloded Zuul for Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs {{!}} https://gerrit.wikimedia.org/r/1145331 {{!}} [[phab:T386691|T386691]]

== 2025-05-13 ==
* 19:27 James_F: Zuul: Upgrade all Quibble 'apitests' jobs from 7.4 to 8.1, for [[phab:T386691|T386691]], [[phab:T328921|T328921]], [[phab:T328922|T328922]]
* 12:35 dcausse: deployment-prep: reindexing wikidata to pickup the &quot;mul&quot; language field ([[phab:T392058|T392058]])
* 08:23 hashar: Update jobs to mute checks for npm packaging files {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1145087/ {{!}} [[phab:T393847|T393847]]

== 2025-05-12 ==
* 16:48 hashar: Updated Jenkins jobs to silence git in ci-src-setup (take 2) {{!}} https://gerrit.wikimedia.org/r/1144596 {{!}} [[phab:T393847|T393847]]
* 16:46 bd808: Reenabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
* 15:55 hashar: Updated Jenkins jobs to silence git in ci-src-setup {{!}} https://gerrit.wikimedia.org/r/1144596 {{!}} [[phab:T393847|T393847]]
* 15:50 bd808: `sudo /usr/local/sbin/clean-stale-puppet-certs --clean` on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud. Attempting to fix a &quot;Found non-revoked Puppet certificates for 1 deleted instances&quot; Prometheus alert.
* 15:28 bd808: Forced puppet run on deployment-etcd05.deployment-prep.eqiad1.wikimedia.cloud to fix Puppet run ([[phab:T393866|T393866]])
* 15:28 bd808: Forced puppet run on deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud to fix Puppet run ([[phab:T393866|T393866]])
* 15:22 bd808: Added `prometheus::instances` and `prometheus::instances_defaults` hiera settings to &quot;deployment-etcd&quot; Prefix Puppet via Horizon ([[phab:T393866|T393866]])
* 12:30 Krinkle: Disable publishing noise from rWSWF, [[phab:T143162|T143162]], [[phab:T267223|T267223]]
* 09:52 hashar: Updating all jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1143972 &quot;Omit noisy `ls` debugging commands when not needed&quot; # [[phab:T282893|T282893]] &amp; [[phab:T393847|T393847]]
* 08:28 hashar: Disabled https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ due to a failure with Etcd/expired certificate # [[phab:T393855|T393855]]
* 08:15 hashar: Updated jobs for &quot;Replace all uses of `$(pwd)` with `$PWD`&quot; {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1143967/
* 07:58 hashar: Disabled https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ due to a failure with Etcd/expired certificate # [[phab:T393855|T393855]]

== 2025-05-08 ==
* 20:28 dancy: Updating buildkitd to v0.21.1 in gitlab-cloud-runners
* 10:58 James_F: Zuul: Support capital first letter of e-mail for Aeywoo in allow list

== 2025-05-07 ==
* 08:52 hashar: Updating Jenkins jobs to Quibble 1.14.1
* 07:03 hashar: Hard rebooted integration-agent-docker-1061 via Horizon, the instance is not reachable by ssh and looks bricked # [[phab:T393542|T393542]]
* 06:58 hashar: Change ssh credentials for integration-agent-docker-1060 integration-agent-docker-1061 and integration-agent-docker-1062 to `key to connect to labs instances set up with role::ci::slave::labs::common` # [[phab:T393543|T393543]]
* 06:57 hashar: Added label `blubber` and `pipelinelib` to integration-agent-docker-1060 integration-agent-docker-1061 and integration-agent-docker-1062 # [[phab:T393543|T393543]]
* 06:41 hashar: integration: bring back integration-agent-docker-1062 , I had it disconnected on April 30 at 6:30am UTC to clean /srv/jenkins/workspace and apparently forgot to put it back online

== 2025-05-06 ==
* 16:16 hashar: restarting CI Jenkins due to a deadlock affecting castor-save-workspace which ends up blocking jobs # [[phab:T353925|T353925]]
* 15:06 hashar: Tag Quibble 1.4.1 @ {{Gerrit|5247438621f802ba9878970b3b34b2d67cefa54c}}

== 2025-05-05 ==
* 14:32 hashar: contint1002 and contint2002: deleted /srv/docker/buildkit following the deletion of /srv/docker/overlay2 earlier today #  [[phab:T393373|T393373]]
* 13:50 hashar: contint1002 and contint2002: deleted /srv/docker/image/overlay2 following the deletion of /srv/docker/overlay2 earlier today #  [[phab:T393373|T393373]]
* 09:45 hashar: Cleared /srv/docker/overlay2 on contint2002
* 09:41 hashar: Cleared /srv/docker/overlay2 on contint1002 (it had bunch of old layers from April/May 2024)

== 2025-05-04 ==
* 13:10 hashar: contint1002: deleted old videos from /srv/jenkins/builds
* 08:59 James_F: Zuul: [AbuseFilter] Add CommunityConfiguration as a Phan dependency, for [[phab:T393240|T393240]]
* 06:33 James_F: Zuul: [mediawiki/extensions/PageImages] Add Scribunto phan dependency, for [[phab:T131911|T131911]]
* 06:33 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Add CLDR dependency

== 2025-05-03 ==
* 10:28 James_F: Zuul: [mediawiki/extensions/PageAssessments] Add Scribunto phan dependency, for [[phab:T380122|T380122]]

== 2025-05-02 ==
* 17:39 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Add Echo as a phan dep
* 12:30 James_F: Zuul: [mediawiki/extensions/CodeEditor] Add BetaFeatures phan dependency, for [[phab:T373711|T373711]]
* 12:18 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst voting again
* 08:43 hashar: Updating Quibble jobs to 1.14.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1140215 {{!}} [[phab:T378797|T378797]] [[phab:T384927|T384927]] [[phab:T386691|T386691]]
* 07:00 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Add cldr as full CI dep too, for [[phab:T391230|T391230]]
* 06:52 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Add cldr as phan dependency, for [[phab:T391230|T391230]]

== 2025-04-30 ==
* 23:46 dancy: Re-enabled https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/
* 18:54 dancy: Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad while Gerrit is down.
* 15:50 hashar: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1140203
* 15:01 hashar: Tagged Quibble 1.14.0 @ {{Gerrit|6d7c736d12daa7ea23b261ede02093f8fe7a83ae}} # [[phab:T378797|T378797]] [[phab:T384927|T384927]] [[phab:T386691|T386691]]
* 06:30 hashar: integration: cleared /srv/jenkins/workspace on integration-agent-docker-1062

== 2025-04-29 ==
* 21:04 mutante: integration-agent-docker-1051.integration - killall -9 ffmpeg - [[phab:T392963|T392963]]
* 20:28 mutante: integration-agent-docker-1048.integration - killall -9 ffpmeg - [[phab:T392963|T392963]]

== 2025-04-28 ==
* 19:01 taavi: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1139536
* 15:49 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/76
* 13:05 James_F: Docker: Bump Node20 and Node22 binaries to latest and cascade

== 2025-04-26 ==
* 00:05 bd808: Punched a hole in the beta cluster network blocks to allow 38.242.176.0/22 through.

== 2025-04-24 ==
* 19:54 thcipriani: deployment-cache-text08: systemctl reload varnish-frontend following puppet run change to /etc/varnish/blocked-nets.inc.vcl
* 19:49 thcipriani: deployment-cache-text08: sudo puppet-run to pick up https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/42c7880be27913c9e841642d9ff3e50deb455e08
* 15:32 bd808: Punched a hole in the beta cluster network blocks to allow 47.144.0.0/12 through. ([[phab:T392534|T392534]])
* 14:41 dancy: Updating runners to v17.9.3 in gitlab-cloud-runners (production)
* 14:34 dancy: Updating runners to v17.9.3 in gitlab-cloud-runners (staging)

== 2025-04-23 ==
* 22:59 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 22:43 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 22:15 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up a huge pile of new blocks ([[phab:T392534|T392534]])
* 22:11 James_F: Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on, for [[phab:T382177|T382177]]
* 21:47 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 21:29 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 20:47 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 17:43 James_F: Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now, for [[phab:T382177|T382177]]
* 16:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/a80e5211100f1cc42e4ae020d4266ea22938eb5a ([[phab:T383097|T383097]])
* 14:25 James_F: Zuul: [wikimedia/portals] Switch to Node 20, for [[phab:T382179|T382179]]

== 2025-04-17 ==
* 10:15 hashar: gerrit: reparented apps.git to All-Archived-Projects.git in order to BLOCK `mediawiki-replication`.  I have also archived all subprojects # [[phab:T392198|T392198]]

== 2025-04-16 ==
* 20:59 bd808: Blocked 193.43.72.0/24 and 14.160.0.0/11 because beta was very, very sad
* 16:02 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst non-voting for now
* 09:20 hashar: integration: restarted integration-puppetserver-01

== 2025-04-15 ==
* 22:02 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting, for [[phab:T368002|T368002]]
* 19:40 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392003|T392003]])
* 18:11 bd808: `bd808@deployment-cache-text08:~$ sudo service varnish-frontend restart` ([[phab:T392003|T392003]])
* 18:06 bd808: `sudo puppet agent -tv` on deployment-cache-text08 to update varnish deny list ([[phab:T392003|T392003]])
* 17:30 bd808: `shutdown -r now` on deployment-mediawiki14. Load has been growing for ~2 days.

== 2025-04-11 ==
* 19:53 James_F: Zuul: [oojs/router] Mark as archived, for [[phab:T391709|T391709]]
* 14:00 hashar: restarted integration-puppetserver: jvm went out of memory

== 2025-04-10 ==
* 23:40 bd808: Removed wikifunctions from deployment-cache prefix puppet's profile::cache::haproxy::available_unified_certificates::server_names. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/6af09ceaa6d261c910fb4b42d7b3e8b8172c8041%5E%21/
* 23:36 bd808: Deleted m.wikifunctions.beta.wmflabs.org, *.wikifunctions.beta.wmflabs.org, and wikifunctions.beta.wmflabs.org DNS records per [[Special:Diff/2292116]]. All 3 were pointing to 185.15.56.36.
* 14:16 hashar: deployment-prep: `profile::mediawiki::php::increase_open_files: True` on https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-mediawiki # [[phab:T389422|T389422]]
* 14:03 James_F: [Beta Cluster] On deployment-deploy04, running DELETE FROM localuser WHERE lu_wiki='wikifunctionswiki'; and DELETE FROM localnames WHERE ln_wiki='wikifunctionswiki'; for [[phab:T391511|T391511]]

== 2025-04-08 ==
* 22:39 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1135128
* 22:15 bd808: Manually deleted 'deployment-wikikube-v127' Magnum cluster template via Horizon. Deletion via OpenTofu has timed out repeatedly.
* 22:08 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1135123
* 22:02 brennen: Updating docker-pkg files on contint primary for [[phab:T383065|T383065]]
* 21:11 James_F: Beta Cluster: Shutting of deployment-docker-wikifunctions01, we decom'ing it.
* 20:44 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1135098

== 2025-04-07 ==
* 17:20 bd808: `service navtiming stop` to halt &quot;Unhandled exception in main loop, restarting consumer&quot; crash loop ([[phab:T391272|T391272]])
* 17:15 bd808: Reboot deployment-webperf21 ([[phab:T391272|T391272]])
* 16:58 bd808: `puppet agent -tv` to catch up with missed puppet runs on deployment-webperf21 ([[phab:T391272|T391272]])
* 16:56 bd808: `rm /var/log/user.log.1` on deployment-webperf21 ([[phab:T391272|T391272]])
* 16:47 bd808: `sudo /usr/local/sbin/clean-stale-puppet-certs --clean` on deployment-puppetserver-1 to clean up dangling certs for deployment-elastic&lt;nowiki&gt;{&lt;/nowiki&gt;09,10,11&lt;nowiki&gt;}&lt;/nowiki&gt;

== 2025-04-04 ==
* 09:42 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 35782 and 35784
* 09:09 hashar: Update tox jobs to default to python 3.9 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1134168
* 08:53 hashar: Updating Quibble jobs to catch up with latest image https://gerrit.wikimedia.org/r/c/integration/config/+/1134167 {{!}} [[phab:T3666646|T3666646]]
* 00:35 thcipriani: integration-agent-docker-1041 marked offline due to /srv disk space
* 00:09 Krinkle: Disable duplicate publishing noise from extension-MediaUploader, ref [[phab:T143162|T143162]], [[phab:T389450|T389450]]

== 2025-04-03 ==
* 15:06 James_F: Zuul: Configure the REL1_44 test and gate pipelines, for [[phab:T390695|T390695]]
* 13:33 James_F: Docker: [quibble-bullseye] Revert MardiaDB to 10.5, for (against) [[phab:T366646|T366646]]
* 13:08 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Publish JS docs

== 2025-04-02 ==
* 13:39 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133383 [[phab:T390754|T390754]]
* 12:36 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133379 https://gerrit.wikimedia.org/r/1133380
* 12:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133373

== 2025-04-01 ==
* 20:46 James_F: Zuul: Swap the branch check to specific release branches, for [[phab:T390754|T390754]] etc.
* 20:34 James_F: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, for [[phab:T366646|T366646]]
* 20:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133238
* 20:09 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133231
* 19:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133221 [[phab:T390754|T390754]]
* 18:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133209 [[phab:T390772|T390772]]
* 16:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133184 [[phab:T390754|T390754]]

== 2025-03-31 ==
* 18:26 dancy: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1132688
* 15:20 James_F: Zuul: [mediawiki/extensions/EmailAuth] Mark as in Wikimedia production, move up, for [[phab:T390437|T390437]]
* 11:08 dcausse: [[phab:T389971|T389971]]: deleting deployment-elastic* VMs in deployment-prep
* 08:24 dcausse: [[phab:T389971|T389971]]: shutting down deployment-elastic* VMs in deployment-prep

== 2025-03-28 ==
* 22:01 Krinkle: Disable duplicate publishing noise from extension-LoginNotify, ref [[phab:T143162|T143162]], [[phab:T390315|T390315]]
* 21:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1130957
* 21:15 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1130957

== 2025-03-27 ==
* 16:28 bd808: Moved Puppet configuration from deployment-cache-text08 to deployment-cache-text prefix Puppet
* 16:05 bd808: `sudo systemctl restart varnish-frontend` on deployment-cache-text08 ([[phab:T390209|T390209]])
* 15:05 bd808: Moved role::acme_chief::cloud from individual instance config to deployment-acme-chief Puppet prefix.
* 00:55 bd808: Removed prefix puppet classes for deployment-acme-chief ([[phab:T390128|T390128]])

== 2025-03-26 ==
* 20:23 inflatador: bking@deployment-prep populating new OpenSearch cluster indices a la https://wikitech.wikimedia.org/w/index.php?title=Search&amp;oldid=2164435#Adding_new_wikis [[phab:T389971|T389971]]
* 17:10 inflatador: bking@deployment-prep reverted an accident replacement of deployment-acme-chief.yaml [[phab:T389971|T389971]]
* 15:04 dancy: Update gitlab-runners to v17.8.4 in gitlab-cloud-runners staging and production.
* 00:30 bd808: Delete parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud service name again ([[phab:T389252|T389252]])

== 2025-03-25 ==
* 21:11 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1130722
* 04:18 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1130729

== 2025-03-24 ==
* 19:35 hashar: Updating Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1130700

== 2025-03-23 ==
* 18:41 James_F: Zuul: Add 0xDeadbeef to CI allowlist
* 18:34 James_F: Zuul: [operations/debs/bdsync] Mark as archived, for [[phab:T377882|T377882]]
* 18:31 James_F: Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency, for [[phab:T386435|T386435]]
* 18:29 James_F: Zuul: [mediawiki/extensions/CategoryWatch] Add Echo CI dependency

== 2025-03-20 ==
* 23:31 bd808: integration: thcipriani added integration-agent-docker-106&lt;nowiki&gt;{&lt;/nowiki&gt;0,1,2&lt;nowiki&gt;}&lt;/nowiki&gt; earlier today ([[phab:T389554|T389554]])
* 22:50 brennen: integration: added jenkins nodes for integration-agent-docker-106&lt;nowiki&gt;{&lt;/nowiki&gt;3,4,5&lt;nowiki&gt;}&lt;/nowiki&gt; with 3 executors per each ([[phab:T389554|T389554]])
* 21:41 brennen: integration: launched integration-agent-docker-106&lt;nowiki&gt;{&lt;/nowiki&gt;3,4,5&lt;nowiki&gt;}&lt;/nowiki&gt; ([[phab:T389554|T389554]])
* 21:25 eileen: civicrm upgraded from {{Gerrit|7b532ad7}} to {{Gerrit|fba4c3d6}}
* 15:13 dancy: Rebooting integration-agent-docker-1046 (Seems to be be inaccessible since February)
* 08:28 taavi: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1129765

== 2025-03-19 ==
* 20:32 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1129364
* 00:12 bd808: Trying the simplest thing that might work by adding a CNAME record for parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud. ([[phab:T389252|T389252]])

== 2025-03-18 ==
* 20:25 bd808: Rebooting deployment-jobrunner05 because things just seem weird ([[phab:T387631|T387631]], [[phab:T387276|T387276]])
* 15:18 sergi0: run CommunityUpdates config schema migration `foreachwikiindblist growthexperiments extensions/CommunityConfiguration/maintenance/migrateConfig.php CommunityUpdates` ([[phab:T387737|T387737]])

== 2025-03-14 ==
* 21:36 Reedy: deployed https://gerrit.wikimedia.org/r/1127982
* 16:55 Lucas_WMDE: manually killed job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81/2928/console which had been stuck since 16:33 UTC, blocking gate-and-submit :(

== 2025-03-13 ==
* 21:29 dancy: Finished gitlab cloud runners k8s production cluster upgrade ([[phab:T388836|T388836]])
* 20:42 dancy: Finished gitlab cloud runners k8s staging cluster upgrade ([[phab:T388836|T388836]])
* 20:09 dancy: Starting gitlab cloud runners k8s production cluster upgrade ([[phab:T388836|T388836]])
* 19:26 dancy: Starting gitlab cloud runners k8s staging cluster upgrade ([[phab:T388836|T388836]])

== 2025-03-11 ==
* 22:54 bd808: Deleted unattached volumes: alert01, db09, deploy03, mwmaint, ores02, parsoid14-srv, prometheus05
* 22:39 bd808: Released unused floating IPs 185.15.56.9 and 185.15.56.97 back to global pool
* 22:08 bd808: Updated mail.beta.wmflabs.org service name to point to 185.15.56.115
* 22:04 bd808: Deleted orphan parsoid-external-ci-access.beta.wmflabs.org. DNS record
* 21:53 bd808: Deleted dangling prometheus-beta.wmcloud.org web proxy
* 21:50 bd808: Deleted dangling w-beta.wmflabs.org web proxy
* 21:42 bd808: Deleted unused &quot;deployment-parsoid&quot; Prefix Puppet configuration
* 20:48 James_F: Docker: [quibble-bullseye-php81 &amp; php81] Use PCRE2 backport from component/php81, for [[phab:T386006|T386006]]
* 13:19 James_F: Zuul: [mediawiki/extensions/ActiveAbstract] Mark as archived, for [[phab:T382069|T382069]]
* 03:54 eileen: civicrm upgraded from {{Gerrit|f2222fcd}} to {{Gerrit|ec20a105}}

== 2025-03-10 ==
* 15:20 James_F: Zuul: [mediawiki/services/servicelib-node] Mark as archived, for [[phab:T388424|T388424]]
* 13:47 hashar: gerrit: removed leftover empty directory `/srv/gerrit/plugins/lfs`. Data have been migrated to `/srv/gerrit/plugins/lfs` as part of moving Gerrit data out of `/`. See [[phab:T333143|T333143]]

== 2025-03-08 ==
* 01:22 James_F: Zuul: [php-session-serializer] Enable PHP 8.4 as voting, for [[phab:T368270|T368270]]

== 2025-03-07 ==
* 21:00 James_F: Zuul: [mediawiki/libs/Shellbox] Enable PHP 8.4 as voting, for [[phab:T386570|T386570]]
* 20:53 James_F: Zuul: [wikipeg] Enable PHP 8.4 as voting, for [[phab:T386570|T386570]]
* 20:07 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.4 as voting, for [[phab:T387806|T387806]]

== 2025-03-05 ==
* 00:21 dancy: Reeanbled beta-scap-sync-world  ([[phab:T166010|T166010]])

== 2025-03-04 ==
* 23:26 dancy: Disabling beta-scap-sync-world for noise reduction while dealing with [[phab:T166010|T166010]]
* 22:10 James_F: Zuul: [mediawiki/services/example-node-api] Mark as archived, for [[phab:T387933|T387933]]
* 01:42 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4, for [[phab:T386570|T386570]]
* 01:13 James_F: Zuul: Add WgevaertWikiBase to CI allowlist
* 01:03 James_F: Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos, for [[phab:T386108|T386108]]

== 2025-02-28 ==
* 18:20 dancy: Upgrading gitlab-runner to v17.7.1 in production gitlab-cloud-runners ([[phab:T386297|T386297]])
* 18:12 dancy: Upgrading gitlab-runner to v17.7.1 in staging gitlab-cloud-runners ([[phab:T386297|T386297]])
* 17:52 dancy: Upgraded scap to 4.138.0 in beta cluster
* 16:43 bd808: Deleted now dangling parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud. DNS record ([[phab:T385849|T385849]])
* 16:40 bd808: Deleted deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T385849|T385849]])
* 16:39 bd808: Deleted parsoid-external-ci-access.wmcloud.org proxy ([[phab:T385849|T385849]])
* 16:37 bd808: Deleted deployment-alert01.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T385849|T385849]])
* 16:36 bd808: Deleted deployment-bastion.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T385849|T385849]])

== 2025-02-27 ==
* 01:11 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1123063 [[phab:T386476|T386476]]

== 2025-02-26 ==
* 20:21 James_F: jforrester@doc1003:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/LdapAuthentication/ #[[phab:T376097|T376097]]
* 20:18 James_F: Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived, for [[phab:T376097|T376097]]
* 13:20 hashar: Updating Quibble jobs to 1.13.0.  &quot;Skip execution upon a success cache hit&quot; which would make some jobs to skip tests entirely when a set of commits/image is known to have previously passed # [[phab:T383243|T383243]] {{!}} dduvall
* 11:06 hashar: Tag Quibble 1.13.0 @ {{Gerrit|0ac128f7bc060c82f11317aabaf78a10b24aeeec}} # [[phab:T383243|T383243]]
* 09:11 hashar: deployment-prep: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1122901 &quot;php: use component/pcre2 when using Php 8.1&quot; to fix php # [[phab:T387276|T387276]]
* 01:55 bd808: `./jjb-update 'integration-quibble-fullrun-*-php81' '*-php81-phan' '*php81*'`
* 01:16 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1122700 [[phab:T386006|T386006]]

== 2025-02-25 ==
* 20:25 James_F: Docker: [php81] Update PHP to 8.1.31-1+wmf11u4, for [[phab:T386006|T386006]]
* 14:07 James_F: Docker: [php81] Upgrade Wikimedia's PHP to 8.1.31-1+wmf11u3 &amp; PCRE to 10.42 for [[phab:T386006|T386006]]

== 2025-02-24 ==
* 01:02 jeena: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/73

== 2025-02-22 ==
* 11:27 taavi: rebooting integration-agent-docker-1047 which thinks it is gerrit

== 2025-02-21 ==
* 22:54 brennen: gitlab: removing expiration date for 14 tokens expiring in 2025 ([[phab:T385930|T385930]])
* 22:36 brennen: gitlab: set require_personal_access_token_expiry and service_access_tokens_expiration_enforced to false

== 2025-02-20 ==
* 20:15 dancy: Updated buildkitd to v0.20.0 in gitlab-cloud-runners ([[phab:T386955|T386955]])
* 20:15 dancy: Updated buildkitd to v0.20.0 in gitlab-cloud-runners

== 2025-02-19 ==
* 21:28 dancy: Reenabled https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/ ([[phab:T386851|T386851]])
* 19:35 dduvall: restarting jenkins to fix git related issues following java update ([[phab:T386755|T386755]])
* 15:47 dancy: Disabled the https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ job to reduce noise while the problem is being debugged.

== 2025-02-18 ==
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1119815
* 16:11 James_F: Zuul: [operations/debs/dnsdist] Revert archival

== 2025-02-13 ==
* 13:57 James_F: Zuul: [mediawiki/extensions/CirrusSearch] Drop WikibaseCirrusSearch dep, for [[phab:T386015|T386015]]

== 2025-02-12 ==
* 17:22 James_F: Zuul: Add User:Michi j to CI allowlist
* 17:21 James_F: Zuul: Add Dragoniez to CI allowlist

== 2025-02-11 ==
* 15:43 James_F: Zuul: Make PHP 8.4 voting on lib repos where it already passes, for [[phab:T386108|T386108]]

== 2025-02-10 ==
* 14:27 James_F: Zuul: Add Bunnypranav to CI allowlist

== 2025-02-08 ==
* 00:07 bd808: Added `profile::maps::osm_master::disable_waterlines_import_timer: false` to deployment-maps prefix hiera ([[phab:T385921|T385921]])

== 2025-02-07 ==
* 22:14 brennen: phab/phorge: replaced mr-widget token in deployed config ([[phab:T385480|T385480]])
* 21:33 bd808: Added `profile::restbase::parsoid_uri: https://phabricator.wikimedia.org/T385902` to deployment-restbase prefix puppet ([[phab:T385902|T385902]])
* 01:34 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117997 to deployment-puppetmaster ([[phab:T385849|T385849]])
* 00:42 bd808: Shutoff deployment-parsoid14 to see if anything breaks/anyone yells ([[phab:T385849|T385849]])

== 2025-02-06 ==
* 23:53 bd808: Updated citoid-beta.wmflabs.org to point to deployment-docker-citoid02
* 23:50 bd808: Deleted beta-prometheus.wmflabs.org; it was pointed to an IP now owned by the mdwikioffline project.
* 23:43 bd808: Deleted recently orphaned spiderpig.wmcloud.org proxy after discussion with dancy
* 16:20 bd808: Rebooted deployment-sessionstore06 ([[phab:T385803|T385803]])
* 12:07 andrewbogott: rebooting all servers for [[phab:T385264|T385264]]

== 2025-02-05 ==
* 19:17 James_F: Zuul: [mediawiki/extensions/DonationInterface] Switch CI from PHP74 to PHP82
* 18:23 James_F: Zuul: [mediawiki/extensions/cldr] Raise FR-special job to REL1_43
* 18:22 James_F: Zuul: [mediawiki/extensions/DonationInterface] Raise FR-special job to REL1_43
* 18:11 James_F: Zuul: [labs/tools/heritage] Fold template into this, only user
* 18:08 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Test in PHP 8.2+ only
* 17:29 James_F: Zuul: [mediawiki/core] Test fundraising branches against PHP 8.2
* 17:19 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Mark as non-prod

== 2025-02-03 ==
* 12:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115782

== 2025-01-30 ==
* 15:12 James_F: Zuul: [mediawiki/extensions/Wikibase] Only inject EntitySchema on 1.43+, for [[phab:T385175|T385175]]
* 01:39 James_F: Zuul: [mediawiki/core] Remove composer variant from wmf branches
* 00:42 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115131

== 2025-01-29 ==
* 18:03 James_F: Zuul: Make FR REL1_43-php82 voting for cldr and FEU
* 17:54 James_F: Zuul: Add FR REL1_43-php82 as experimental to other extensions
* 17:40 James_F: Zuul: [mediawiki/extensions/cldr] Add FR REL1_43-php82 as experimental
* 17:40 James_F: Zuul: [mediawiki/extensions/cldr] Re-enable FR-tech job as voting, passes fine
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115064
* 16:33 hashar: gerrit: marked all legacy Puppet modules as read-only ( https://gerrit.wikimedia.org/r/admin/repos/q/filter:operations/puppet/ ) and removed the associated GitHub mirrors that existed for some of them

== 2025-01-28 ==
* 17:46 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1113550 ([[phab:T383337|T383337]])
* 17:38 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1113549 ([[phab:T383337|T383337]])
* 10:07 hashar: Manually cleaned integration-agent-docker-1043

== 2025-01-27 ==
* 18:17 hashar: Cleaned disk on integration-agent-docker-1051

== 2025-01-25 ==
* 09:20 taavi: reloading zuul for https://gerrit.wikimedia.org/r/1113739

== 2025-01-24 ==
* 21:44 James_F: Revert &quot;Zuul: Switch Fundraising jobs to REL1_43&quot;

== 2025-01-23 ==
* 16:31 dancy: Updating production gitlab-cloud-runners to v17.6.1
* 16:23 dancy: Updating staging gitlab-cloud-runners to v17.6.1

== 2025-01-22 ==
* 18:14 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add Wikibase as a phan dependency

== 2025-01-20 ==
* 09:55 hashar: Updating Quibble jobs to enable success cache experiment - [[phab:T383243|T383243]]
* 08:20 hashar: Updating all Jenkins jobs to update Quibble to 1.12.0

== 2025-01-17 ==
* 16:59 dduvall: Building Docker images for Quibble 1.12.0
* 15:00 hashar: Building Docker images for Quibble 1.12.0
* 12:56 hashar: Tag Quibble 1.12.0 @ {{Gerrit|633099ead3ec72180e7890e1980074b4fde56c26}} # [[phab:T365978|T365978]], [[phab:T383243|T383243]]

== 2025-01-14 ==
* 17:14 brennen: integration project: create integration-agent-docker-1059 for [[phab:T383254|T383254]]
* 16:50 brennen: integration project: create integration-agent-docker-1058 for [[phab:T383254|T383254]]

== 2025-01-10 ==
* 15:55 dancy: Updating gitlab-cloud-runners (prod) to v17.5.5 ([[phab:T383263|T383263]])
* 15:49 dancy: Updating gitlab-cloud-runners (staging) to v17.5.5

== 2025-01-09 ==
* 22:20 brennen: gitlab: Feature.enable(:kubernetes_agent_protected_branches) - https://docs.gitlab.com/ee/user/clusters/agent/ci_cd_workflow.html#restrict-access-to-the-agent-to-protected-branches
* 18:08 James_F: Docker: [node22] Update Node to v22.13.0, &amp; switch base image to bookworm, for [[phab:T383337|T383337]]
* 17:01 James_F: Docker: [node20] Update Node to v20.18.1, &amp; switch base image to bookworm, for [[phab:T383337|T383337]]
* 15:13 James_F: Docker: [sury-php] Re-platform to bookworm

== 2025-01-08 ==
* 22:07 hashar: castor: deleting potentially corrupted npm cache. On integration-castor05: sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/&lt;nowiki&gt;{&lt;/nowiki&gt;wmf-quibble-selenium-php74,quibble-vendor-mysql-php74-selenium&lt;nowiki&gt;}&lt;/nowiki&gt;/npm  # [[phab:T383237|T383237]]

== 2025-01-07 ==
* 22:07 hashar: Deleted /srv/zuul/git/operations/dumps/dcat on both contint1002 and contint2002 # [[phab:T157818|T157818]]
* 19:00 bd808: `/usr/local/sbin/clean-stale-puppet-certs --clean` ([[phab:T383153|T383153]])
* 18:53 taavi: taavi@deployment-puppetserver-1:~$ sudo puppetserver ca clean --certname maps-master01.maps-experiments.eqiad1.wikimedia.cloud # [[phab:T383153|T383153]]
* 18:50 taavi: taavi@deployment-puppetserver-1:~$ sudo puppet node clean geoshapes.maps-experiments.eqiad1.wikimedia.cloud # [[phab:T383153|T383153]]
* 18:30 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance deployment-etcd04
* 18:30 bd808@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance deployment-etcd04
* 14:48 hashar: Manually renamed wikibase-daily-npm-audit-daily-node18-npmaudit to node20 variant and refresh the config with JJB
* 14:33 James_F: Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master

== 2025-01-06 ==
* 20:16 andrewbogott: removed the (non-existent?) role::mw_rc_irc from puppet config for deployment-ircd03.deployment-prep.eqiad1.wikimedia.cloud
* 19:35 bd808: Manually generated missing en_US.UTF-8 locale on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T361381|T361381]])
* 19:32 bd808: Added `postgresql::postgis::postgresql_postgis_package: postgresql-15-postgis-3` to deployment-maps Prefix Puppet to work around default parameter problem ([[phab:T361381|T361381]])
* 19:31 bd808: Issued new Puppet cert for deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T361381|T361381]])
* 19:27 bd808: Added `postgresql::postgis::postgresql_postgis_package: ignored` to deployment-maps Prefix Puppet to work around default parameter problem ([[phab:T361381|T361381]])
* 19:15 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 ([[phab:T382709|T382709]])
* 19:11 bd808: Added placeholders for `graphite_host` and `statsd` to deployment-webperf Prefix Puppet
* 18:53 bd808: Fixed missing profile::swift::global_account_keys::&lt;nowiki&gt;{&lt;/nowiki&gt;codfw, eqiad&lt;nowiki&gt;}&lt;/nowiki&gt; placeholders breaking deployment-ms-* puppet runs
* 18:38 bd808: Fixed incorrect deployment-restbase prefix puppet setting that was causing puppet run failures
* 18:19 bd808: Issued a new Puppet client cert for traindev01.deployment-prep.eqiad1.wikimedia.cloud
* 14:58 James_F: Zuul: Drop CI for REL1_41 branch, now EOL per [[phab:T376550|T376550]]
* 09:03 hashar: gerrit: flushed diff_intraline, diff_summary, gerrit_file_diff and git_file_diff caches after having turned on diff3 style # [[phab:T359821|T359821]]

== 2025-01-02 ==
* 11:27 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1105679 # [[phab:T374113|T374113]]


{{SAL-archives/Release Engineering}}

&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>euw8qzbeumyrymae1r69gd8z7yz1pgf</sha1>
    </revision>
    <revision>
      <id>2320898</id>
      <parentid>2320897</parentid>
      <timestamp>2025-07-07T08:45:31Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>hashar: gerrit: change operations/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; | T390719</comment>
      <origin>2320898</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="56217" sha1="lhwg04gx8qy78wjye4z8ybpieop8pga" xml:space="preserve">== 2025-07-07 ==
* 08:45 hashar: gerrit: change operations/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 08:45 hashar: gerrit: change mediawiki/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]

== 2025-07-04 ==
* 21:45 Krinkle: [[phab:T289318|T289318]]: Change stream_config_uri in Hiera (Horizon instance config for  deployment-eventgate-4 and deployment-eventstreams-2 ) from https://meta.wikimedia.beta.wmflabs.org/w/api.php?action=streamconfigs to https://meta.wikimedia.beta.wmcloud.org/w/api.php?action=streamconfigs
* 21:45 Krinkle: [[phab:T289318|T289318]]: Change profile::cache::varnish::frontend::fe_vcl_config/static_host in Hiera (Horizon puppet prefix for cache-text and cache-upload) from  en.wikipedia.beta.wmflabs.org to en.wikipedia.beta.wmcloud.org
* 21:41 Krinkle: Change profile::docker::runner::service_defs/mediawiki-services-cxserver/mwapi_req/host in  Horizon (Hiera puppet prefix) from en.wikipedia.beta.wmflabs.org to en.wikipedia.beta.wmcloud.org. [[phab:T289318|T289318]]
* 21:39 Krinkle: Change profile::docker::runner::service_defs/mediawiki-services-push-notifications/mwapi_req/host in  Horizon (Hiera puppet prefix) from meta.wikimedia.beta.wmflabs.org to meta.wikimedia.beta.wmcloud.org. [[phab:T289318|T289318]]
* 13:49 hashar: gerrit: deleted project glam/gwtoolset {{!}} Created October 11st 2012 and has never been used
* 13:24 hashar: gerrit: changed `All-Projects` default submit strategy to `Rebase if Necessary`. Does not affect mediawiki/* or operations/* among others # [[phab:T390719|T390719]]

== 2025-07-02 ==
* 21:41 Krinkle: [[phab:T289318|T289318]] - Change service::catalog probes for mw-api-int in Horizon prefix Puppet from en.wikipedia.beta.wmflabs.org/w/api.php to en.wikipedia.beta.wmcloud.org/w/api.php
* 21:38 Krinkle: [[phab:T289318|T289318]] - Change profile::mail::mx::verp_bounce_post_url in Horizon prefix puppet, from https://meta.wikimedia.beta.wmflabs.org/w/api.php to https://meta.wikimedia.beta.wmcloud.org/w/api.php.
* 17:33 hashar: Reloaded Zuul for &quot;Drop generic ruby rake jobs&quot; https://gerrit.wikimedia.org/r/c/integration/config/+/1165947/
* 14:51 hashar: Zuul: Upgrade translatewiki-ruby* from 2.5 to 2.7, for [[phab:T335765|T335765]]
* 14:13 James_F: Zuul: Upgrade ooui-ruby* from 2.5 to 2.7, for [[phab:T335765|T335765]]
* 07:47 hashar: gerrit: ssh -p 29418 gerrit.wikimedia.org rename-project operations/debs/wmf-sre-laptop operations/debs/wmf-laptop # [[phab:T365985|T365985]]

== 2025-07-01 ==
* 10:32 hashar: gerrit: deleted secrets/wikimetrics , a 2016 experiment to hold credentials for deployment purpose # [[phab:T219334|T219334]]
* 08:21 hashar: gerrit: archived https://gerrit.wikimedia.org/g/qrpedia Latest source code is elsewhere {{!}} [[phab:T244135|T244135]]
* 07:41 hashar: Disabled CI for REL1_42 # [[phab:T389313|T389313]]

== 2025-06-30 ==
* 22:09 bd808: Blocked 4 Class C networks with &gt;1000 hits in the last 100,000 Beta Cluster requests
* 21:40 bd808: Unblocked 46.28.80.0/21 at CDN edge ([[phab:T398124|T398124]])
* 20:17 bd808: Upgraded haproxy to 2.8.14-1~bpo11+1 on deployment-cache-text08 ([[phab:T398176|T398176]])
* 20:13 bd808: Upgraded haproxy to 2.8.14-1~bpo11+1 on deployment-cache-upload08 ([[phab:T398176|T398176]])
* 20:03 bd808: Remove `profile::cache::haproxy::version: haproxy26` from deployment-cache Prefix Puppet ([[phab:T398176|T398176]])
* 17:31 hashar: gerrit: marked read-only all operations/debs/contenttranslation/apertium* repositories. Untouched since 2020.
* 16:37 hashar: gerrit: change wikimedia/fundraising/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:57 hashar: gerrit: change labs/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:37 hashar: gerrit: change mediawiki/libs/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:31 hashar: gerrit: change performance/* submit strategy to &quot;Rebase if Necessary&quot; and &quot;Allow content merge&quot; {{!}} [[phab:T390719|T390719]]
* 13:28 hashar: gerrit: deleted videojs-resolution-switcher and videojs-responsive-layout , forks of other projects with no local modifications/changes.

== 2025-06-27 ==
* 14:12 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1164451

== 2025-06-26 ==
* 14:49 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1164197 ([[phab:T397922|T397922]])
* 14:43 dancy: Updated gitlab-cloud-runners to gitlab-runner v17.11.3 ([[phab:T397899|T397899]])
* 10:55 urbanecm: deployment-prep: Run `foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/importOresTopics.php --count=20000 --verbose` ([[phab:T393684|T393684]])

== 2025-06-25 ==
* 21:16 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1163883/1 to deployment-puppetserver-1 ([[phab:T397877|T397877]])
* 20:24 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/3 to deployment-puppetserver-1 ([[phab:T397872|T397872]])
* 18:19 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1137013/2 to deployment-puppetserver-1 ([[phab:T397717|T397717]])
* 17:05 thcipriani: Upgrading scap to 4.182.0 in beta cluster
* 08:55 hashar: jenkins: updated job publish-to-doc to use label productionAgents rather than contint1002 # [[phab:T397815|T397815]]
* 08:52 hashar: jenkins: updated jobs fail-archived-repositories, train-deploy-notes and trigger-* to use label productionAgents rather than contint1002 # [[phab:T397815|T397815]]
* 02:19 Krinkle: Add mapping for performance.wikimedia.beta.wmcloud.org to profile::trafficserver::backend::mapping_rules in Hiera under deployment-cache-text prefix. Same mapping as the wmflabs version. [[phab:T289318|T289318]]

== 2025-06-23 ==
* 16:41 greg-g: removed 2fa from XenoRyet, confirmed on video call
* 16:05 dancy: Ran `docker run --rm -it --network gitlab-runner --entrypoint buildctl docker-registry.wikimedia.org/repos/releng/buildkit:wmf-v0.22.0 --addr buildkitd:1234 prune` on `runner-1025.gitlab-runners.eqiad1.wikimedia.cloud
* 07:20 James_F: Zuul: [mediawiki/extensions/EventLogging] Add CodeEditor Phan dependency, for [[phab:T346540|T346540]]

== 2025-06-22 ==
* 21:42 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1162179

== 2025-06-21 ==
* 02:54 Krinkle: Updating docker-pkg files on contint primary for  https://gerrit.wikimedia.org/r/1162106

== 2025-06-20 ==
* 18:57 dduvall: ran `helm --namespace gitlab-runner uninstall docker-hub-mirror` to fix helm state. reapplying production cluster configuration
* 18:41 dduvall: deleted docker-hub-mirror statefulset and admission controller deployment. reapplying production cluster configuration
* 18:18 dduvall: seeing numerous image pull errors in gitlab-cloud-runner cluster

== 2025-06-19 ==
* 09:38 sergi0: deployment-prep: GrowthExperiments config migration `foreachwiki extensions/CommunityConfiguration/maintenance/migrateConfig.php GrowthSuggestedEdits` — [[phab:T393771|T393771]]
* 09:18 urbanecm: deployment-prep: Update changeprop config perhttps://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1161443 using [[wikitech:Changeprop#To_deployment-prep]] ([[phab:T394958|T394958]]; this time actually changing the beta config)
* 09:10 urbanecm: deployment-prep: Update changeprop config per https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1150699 using [[wikitech:Changeprop#To_deployment-prep]] ([[phab:T394958|T394958]])

== 2025-06-18 ==
* 23:26 bd808: Blocked 128.241.0.0/16 &quot;NTT America&quot; network. ([[phab:T397378|T397378]])
* 22:10 bd808: Blocked 202.76.160.0/20 &quot;Huawei-Cloud-SG&quot; network. ([[phab:T397378|T397378]])
* 22:02 bd808: Blocked 146.174.160.0/19 &quot;Huawei-Cloud-SG&quot; network. ([[phab:T397378|T397378]])
* 18:19 bd808: `docker system prune --all` on runner-1023.gitlab-runners.eqiad1.wikimedia.cloud
* 13:10 James_F: Zuul: Add EggRoll97 to CI allowlist
* 13:08 James_F: Zuul: Add James E. Blair to CI allowlist
* 13:06 James_F: Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template
* 04:14 Krinkle: Fix profile::trafficserver::backend::mapping_rules in deployment-cache-text to include `rb-mw-mangling-beta.lua` as otherwise w.beta.wmcloud.org serves 404 Domain Not Configured, ref [[phab:T289318|T289318]], [[phab:T396012|T396012]]
* 04:13 Krinkle: Fix profile::trafficserver::backend::mapping_rules in deployment-cache-upload to include `rb-mw-mangling-beta.lua` as otherwise w.beta.wmcloud.org serves 404 Domain Not Configured, ref [[phab:T289318|T289318]], [[phab:T396012|T396012]]
* 04:10 Krinkle: Change shortener_domain in deployment-cache-text prefix from `w-beta.wmflabs.org` to `w.beta.wmcloud.org`, to apply VCL normalization for w.wiki in Beta, ref [[phab:T289318|T289318]], [[phab:T396012|T396012]]

== 2025-06-16 ==
* 15:15 James_F: Docker: [quibble-bullseye] Add the MariaDB binaries to our path [[phab:T366646|T366646]]
* 14:32 James_F: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again, for [[phab:T366646|T366646]]

== 2025-06-13 ==
* 15:50 James_F: Docker: Drop php-ast image, now unused, for [[phab:T396312|T396312]]
* 15:48 James_F: Zuul: Drop broken composer-coverage-patch job from the two repos using it

== 2025-06-12 ==
* 20:41 bd808: `sudo service varnish-frontend restart` on deployment-cache-text08 to pick up blocked_nets changes ([[phab:T394881|T394881]])
* 20:28 bd808: `sudo service varnish-frontend restart` on deployment-cache-text08 to pick up blocked_nets changes ([[phab:T396748|T396748]])
* 20:15 bd808: Added `profile::memcached::firewall_srange: ~` to deployment-memc Puppet prefix ([[phab:T396732|T396732]])
* 16:24 James_F: Docker: Cascade uses of php* with new php-ast inline build, for [[phab:T396312|T396312]]
* 15:23 dancy: Upgraded gitlab-cloud-runners to v17.10.2 ([[phab:T396701|T396701]])
* 15:04 James_F: Docker: [node-test-brower-php*-composer] Build php-ast inline, for [[phab:T396312|T396312]]
* 14:50 James_F: Docker: [php*] Build php-ast with the exact same PHP version, for [[phab:T396312|T396312]]

== 2025-06-10 ==
* 22:53 James_F: Zuul: [css-sanitizer] Add coverage reporting
* 20:02 brennen: Updating buildkitd to v0.22.0 in gitlab-cloud-runners ([[phab:T394931|T394931]])
* 14:37 James_F: Zuul: [maps/*] Mark all as archived
* 13:33 sergi0: run migration in GrowthSuggestedEditsSchema `foreachwikiindblist growthexperiments extensions/CommunityConfiguration/maintenance/migrateConfig.php GrowthSuggestedEdits` [[phab:T395383|T395383]]
* 13:31 sergi0: set version in GrowthSuggestedEdits  schema `foreachwiki extensions/CommunityConfiguration/maintenance/setVersionData.php GrowthSuggestedEdits 1.0.0`
* 11:35 James_F: jforrester@integration-castor05:/srv/castor$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-rundoc/ # [[phab:T396426|T396426]]

== 2025-06-09 ==
* 15:01 James_F: Zuul: [labs/tools/WdTmCollab] Add tox job CI, for [[phab:T396349|T396349]]
* 14:25 James_F: Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Mark as archived, for [[phab:T396311|T396311]]
* 14:16 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Test on PHP 8.4, for [[phab:T386570|T386570]]

== 2025-06-08 ==
* 18:14 James_F: Zuul: [mediawiki/extensions/Echo] Remove EventLogging
* 18:12 James_F: Zuul: Fold extension-quibble-php81-or-later template into extension-quibble
* 18:04 James_F: Zuul: [mediawiki/extensions/SemanticVersion] Add basic CI

== 2025-06-06 ==
* 14:37 jnuche: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/79

== 2025-06-05 ==
* 23:21 thcipriani: update scap in beta to 4.171.0 to match prod
* 20:44 James_F: Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base, archive repo's CI, for [[phab:T354310|T354310]]
* 20:20 bd808: Added `profile::memcached::firewall_src_sets: ~` to deployment-memc prefix puppet ([[phab:T396109|T396109]])
* 19:03 Krinkle: Update profile::tlsproxy::envoy::cfssl_options under deployment-mediawiki in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary. ref [[phab:T289318|T289318]]
* 18:26 James_F: Docker: Re-build PHP images with php-uuid (and incidentally bump versions), for [[phab:T373752|T373752]]
* 17:14 James_F: Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81
* 17:10 James_F: Docker: [phpmetrics] Migrate parent image from php74 to php81
* 17:10 James_F: Where will Abstract Content go?
* 17:07 James_F: Zuul: [mediawiki/extensions/WikimediaMaintenance] Add dependencies, for [[phab:T58074|T58074]]
* 16:39 James_F: Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Use a template for CI
* 16:37 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Stop testing in PHP 7.4
* 16:36 James_F: Zuul: [labs/tools/heritage] Raise PHP testing from 7.4 to 8.1
* 16:34 James_F: Zuul: Stop testing most libraries and tools in PHP 7.4
* 16:28 James_F: Zuul: Stop testing PHP extensions with PHP 7.4
* 16:26 James_F: Zuul: [integration/quibble] Stop testing in PHP 7.4, for [[phab:T328921|T328921]] and [[phab:T328922|T328922]]
* 16:23 James_F: Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4
* 16:21 James_F: Zuul: [operations/mediawiki-config] Stop testing in PHP 7.4
* 16:09 James_F: Zuul: Drop all PHP 7.4 testing for MediaWiki things, for [[phab:T328921|T328921]] and [[phab:T328922|T328922]]
* 04:46 Krinkle: gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$  Cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1153764, ref [[phab:T289318|T289318]]
* 03:58 Krinkle: Update profile::cache::haproxy::available_unified_certificates under deployment-cache in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary. Remove `*.zero.wikipedia.beta.wmflabs.org` which wasn't responding/didn't work anymore. ref [[phab:T289318|T289318]]
* 03:34 Krinkle: Update profile::acme_chief::certificates under deployment-acme-chief prefix in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary (wikipedia and wikivoyage were already there), ref [[phab:T289318|T289318]]
* 03:34 Krinkle: Update profile::acme_chief::certificates under deployment-acme-chief prefix in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary (wikipedia and wikivoyage were already there)
* 00:32 Krinkle: Add `TXT *.wikimedia.beta.wmcloud.org. &quot;v=spf1 -all&quot;` to match beta.wmflabs.org DNS (ref [[phab:T289318|T289318]], changing email is out of scope for now, but might as well add the DNS records).
* 00:22 Krinkle: Adding missing DNS entries under beta.wmcloud.org. There was already: *.wikipedia, *.m.wikimedia, *.wikivoyage, *.m.wikivoyage (for [[phab:T355281|T355281]]). Adding: wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary, wikidata, upload ([[phab:T289318|T289318]]).

== 2025-06-04 ==
* 21:27 James_F: Zuul: [mediawiki/extensions/Springboard] Add basic CI, for [[phab:T395981|T395981]]
* 12:10 lucaswerkmeister: lucaswerkmeister@deployment-deploy04:~$ mwscript createAndPromote commonswiki --interface-admin --force 'Lucas Werkmeister' # w-beta.wmflabs.org/mt

== 2025-06-03 ==
* 23:59 James_F: Zuul: [mediawiki/services/&lt;some&gt;] Upgrade test suite to Node 24 &amp; 22, for [[phab:T395926|T395926]]
* 23:56 James_F: Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22, for [[phab:T395926|T395926]]
* 23:56 James_F: Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22, for [[phab:T395926|T395926]]
* 23:55 James_F: Zuul: [oojs/*i] Upgrade test suite to Node 24 and Node 22, for [[phab:T395926|T395926]]
* 23:53 James_F: Zuul: [wikimedia/portals/deploy] Drop tests, this repo isn't testable
* 23:20 James_F: Zuul: Provide experimental Node 24 jobs where Node 22 ones exist, for [[phab:T395926|T395926]]
* 17:09 bd808: Forced puppet run on deployment-webperf21 to pick up Kafka config changes ([[phab:T391273|T391273]])
* 17:08 bd808: Manually expanded (duplicated) jumbo-eqiad and main-eqiad aliases in kafka_clusters hiera config ([[phab:T391273|T391273]])
* 17:04 bd808: Added jumbo-eqiad and main-eqiad aliases to kafka_clusters hiera config ([[phab:T391273|T391273]])
* 16:00 James_F: Docker: Provide initial Node 24 images, for [[phab:T395923|T395923]]
* 09:53 TheresNoTime: `samtar@deployment-cache-text08:~$ sudo service varnish-frontend restart` for [[phab:T395808|T395808]]
* 09:52 TheresNoTime: `samtar@deployment-cache-text08:~$ sudo -i puppet agent -tv` for [[phab:T395808|T395808]]

== 2025-06-02 ==
* 14:37 James_F: Zuul: Add Matrix to CI allowlist
* 14:37 James_F: Zuul: [operations/software/gerrit/plugins/events-wikimedia] mark as archived, for [[phab:T304947|T304947]]
* 14:36 James_F: Zuul: [mediawiki/extensions/CookieConsent] Add basic CI
* 13:45 hashar: Updating  Jenkins jobs for &quot;drop obsolete creation of log &amp; src dirs&quot; {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1152702

== 2025-05-30 ==
* 22:16 thcipriani: killed 1000s of zuul merger jobs via https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions for parsoid, wikibase, and core
* 21:20 bd808: Poked hole in blocked_nets for 188.214.8.0/21 ([[phab:T395709|T395709]])
* 09:43 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 57273 and 57274

== 2025-05-29 ==
* 22:18 bd808: Submitted WikimediaDebug v3.1.0 to addons.mozilla.org for review ([[phab:T395190|T395190]], [[phab:T315111|T315111]])
* 22:12 bd808: Submitted WikimediaDebug v3.1.0 to Chrome Web Store for review ([[phab:T395190|T395190]], [[phab:T315111|T315111]])

== 2025-05-28 ==
* 20:27 James_F: Zuul: [mediawiki/extensions/ArticleSummaries] Promote to Wikimedia production, for [[phab:T393940|T393940]]
* 13:15 James_F: [Beta Cluster] On deployment-deploy04, running DELETE FROM localuser WHERE lu_wiki='en_rtlwiki'; and DELETE FROM localnames WHERE ln_wiki='en_rtlwiki'; as part of closing the wiki
* 12:30 James_F: Zuul: Add an explanatory note to bluespice template that we skip non-LTSes

== 2025-05-24 ==
* 21:52 Krinkle: Disable publishing notifs on Phab tasks from extension-Chart mirror, [[phab:T143162|T143162]], [[phab:T272803|T272803]]

== 2025-05-23 ==
* 18:36 James_F: Zuul: [mediawiki/core] Restore node testing for release branches, for [[phab:T395141|T395141]]
* 17:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1149705

== 2025-05-22 ==
* 21:15 bd808: Forced Puppet run and restarted varnins-frontend on deployment-cache-upload08 to pick up new config ([[phab:T393404|T393404]])
* 21:12 bd808: Forced Puppet run and restarted varnins-frontend on deployment-cache-text08 to pick up new config ([[phab:T393404|T393404]])
* 21:09 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143602 ([[phab:T393404|T393404]])
* 21:09 bd808: Added `block_help: &quot;see https://wikitech.wikimedia.org/wiki/Beta/Blocked_help for more information.&quot;` under `profile::cache::varnish::frontend::fe_vcl_config` in both deployment-cache-text and deployment-cache-upload Prefix Puppet ([[phab:T393404|T393404]])
* 20:11 brennen: devtools: phorge: test deploying work/merge-phorge-2024.35 changes
* 17:25 bd808: `./jjb-update 'selenium-daily-beta*-MediaWiki'` to deploy updates to selenium-daily-beta-MediaWiki and selenium-daily-betacommons-MediaWiki failure notifications ([[phab:T394551|T394551]])
* 14:45 dancy: Upgrade gitlab-runner to v17.10.1 in gitlab-cloud-runner (staging and production) [[phab:T394953|T394953]]
* 11:39 hashar: Triggered replication of mediawiki/extensions/BlueSpiceSmartlist and mediawiki/extensions/BlueSpiceSmartList to fix https://github.com/wikimedia/mediawiki-extensions-BlueSpiceSmartlist {{!}} [[phab:T394903|T394903]]
* 11:37 hashar: gerrit: changed parent of mediawiki/extensions/BlueSpiceSmartlist (lower case L) to All-Archived-Projects to prevent it from being replicated to GitHub {{!}} [[phab:T394903|T394903]]

== 2025-05-21 ==
* 07:24 hashar: restarted Gerrit on gerrit1003
* 07:18 hashar: restarted Jenkins on contint1002

== 2025-05-20 ==
* 17:51 bd808: Open CDN edge blocks to allow traffic from 190.217.20.32/28
* 17:13 dancy: Restarting Jenkins on contint1002
* 16:27 James_F: Docker: [quibble-bullseye-php81-coverage]: Fix clover-edit for py39
* 14:30 James_F: Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15
* 14:28 hashar: integration: cleared Docker build cache on integration-agent-docker-1052 and integration-agent-docker-1061
* 13:49 James_F: Docker: Provide quibble-bullseye-php81-coverage

== 2025-05-19 ==
* 15:48 James_F: Zuul: Switch primary master branch testing to PHP 8.1, not 7.4
* 15:45 James_F: Zuul: Switch / remove any experimental testing to PHP 8.1, not 7.4
* 15:39 James_F: Zuul: Switch REL1_39 branch testing to PHP 8.1, not 7.4
* 15:37 James_F: Zuul: Switch all wmf branch testing to PHP 8.1, not 7.4
* 13:25 James_F: Zuul: Simplify the regular Quibble job name to drop 'noselenium'
* 13:24 James_F: jjb: Simplify the regular Quibble job name to drop 'noselenium'
* 12:18 hashar: integration: cleaned Docker build cache on integration-agent-docker-1045
* 09:26 hashar: integration: cleaned Docker build cache on integration-agent-docker-1040

== 2025-05-16 ==
* 16:57 James_F: Zuul: Split Quibble jobs into selenium-only and non-selenium for skins

== 2025-05-15 ==
* 21:22 bd808: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1146722
* 13:54 James_F: Zuul: [mediawiki/extensions/Realnames] Use vendor quibble, not composer
* 09:34 codders: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1146520

== 2025-05-14 ==
* 21:31 bd808: Restarted varnish-frontend on deployment-cache-text08 to pick up blocked_nets changes ([[phab:T394311|T394311]])
* 16:06 hashar: Updating jobs for &quot;jjb: silence some shell blocks in macro-docker.yaml&quot; {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1145090 {{!}} [[phab:T393847|T393847]]
* 13:43 hashar: Reloded Zuul for Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs {{!}} https://gerrit.wikimedia.org/r/1145331 {{!}} [[phab:T386691|T386691]]

== 2025-05-13 ==
* 19:27 James_F: Zuul: Upgrade all Quibble 'apitests' jobs from 7.4 to 8.1, for [[phab:T386691|T386691]], [[phab:T328921|T328921]], [[phab:T328922|T328922]]
* 12:35 dcausse: deployment-prep: reindexing wikidata to pickup the &quot;mul&quot; language field ([[phab:T392058|T392058]])
* 08:23 hashar: Update jobs to mute checks for npm packaging files {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1145087/ {{!}} [[phab:T393847|T393847]]

== 2025-05-12 ==
* 16:48 hashar: Updated Jenkins jobs to silence git in ci-src-setup (take 2) {{!}} https://gerrit.wikimedia.org/r/1144596 {{!}} [[phab:T393847|T393847]]
* 16:46 bd808: Reenabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs
* 15:55 hashar: Updated Jenkins jobs to silence git in ci-src-setup {{!}} https://gerrit.wikimedia.org/r/1144596 {{!}} [[phab:T393847|T393847]]
* 15:50 bd808: `sudo /usr/local/sbin/clean-stale-puppet-certs --clean` on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud. Attempting to fix a &quot;Found non-revoked Puppet certificates for 1 deleted instances&quot; Prometheus alert.
* 15:28 bd808: Forced puppet run on deployment-etcd05.deployment-prep.eqiad1.wikimedia.cloud to fix Puppet run ([[phab:T393866|T393866]])
* 15:28 bd808: Forced puppet run on deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud to fix Puppet run ([[phab:T393866|T393866]])
* 15:22 bd808: Added `prometheus::instances` and `prometheus::instances_defaults` hiera settings to &quot;deployment-etcd&quot; Prefix Puppet via Horizon ([[phab:T393866|T393866]])
* 12:30 Krinkle: Disable publishing noise from rWSWF, [[phab:T143162|T143162]], [[phab:T267223|T267223]]
* 09:52 hashar: Updating all jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1143972 &quot;Omit noisy `ls` debugging commands when not needed&quot; # [[phab:T282893|T282893]] &amp; [[phab:T393847|T393847]]
* 08:28 hashar: Disabled https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ due to a failure with Etcd/expired certificate # [[phab:T393855|T393855]]
* 08:15 hashar: Updated jobs for &quot;Replace all uses of `$(pwd)` with `$PWD`&quot; {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1143967/
* 07:58 hashar: Disabled https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ due to a failure with Etcd/expired certificate # [[phab:T393855|T393855]]

== 2025-05-08 ==
* 20:28 dancy: Updating buildkitd to v0.21.1 in gitlab-cloud-runners
* 10:58 James_F: Zuul: Support capital first letter of e-mail for Aeywoo in allow list

== 2025-05-07 ==
* 08:52 hashar: Updating Jenkins jobs to Quibble 1.14.1
* 07:03 hashar: Hard rebooted integration-agent-docker-1061 via Horizon, the instance is not reachable by ssh and looks bricked # [[phab:T393542|T393542]]
* 06:58 hashar: Change ssh credentials for integration-agent-docker-1060 integration-agent-docker-1061 and integration-agent-docker-1062 to `key to connect to labs instances set up with role::ci::slave::labs::common` # [[phab:T393543|T393543]]
* 06:57 hashar: Added label `blubber` and `pipelinelib` to integration-agent-docker-1060 integration-agent-docker-1061 and integration-agent-docker-1062 # [[phab:T393543|T393543]]
* 06:41 hashar: integration: bring back integration-agent-docker-1062 , I had it disconnected on April 30 at 6:30am UTC to clean /srv/jenkins/workspace and apparently forgot to put it back online

== 2025-05-06 ==
* 16:16 hashar: restarting CI Jenkins due to a deadlock affecting castor-save-workspace which ends up blocking jobs # [[phab:T353925|T353925]]
* 15:06 hashar: Tag Quibble 1.4.1 @ {{Gerrit|5247438621f802ba9878970b3b34b2d67cefa54c}}

== 2025-05-05 ==
* 14:32 hashar: contint1002 and contint2002: deleted /srv/docker/buildkit following the deletion of /srv/docker/overlay2 earlier today #  [[phab:T393373|T393373]]
* 13:50 hashar: contint1002 and contint2002: deleted /srv/docker/image/overlay2 following the deletion of /srv/docker/overlay2 earlier today #  [[phab:T393373|T393373]]
* 09:45 hashar: Cleared /srv/docker/overlay2 on contint2002
* 09:41 hashar: Cleared /srv/docker/overlay2 on contint1002 (it had bunch of old layers from April/May 2024)

== 2025-05-04 ==
* 13:10 hashar: contint1002: deleted old videos from /srv/jenkins/builds
* 08:59 James_F: Zuul: [AbuseFilter] Add CommunityConfiguration as a Phan dependency, for [[phab:T393240|T393240]]
* 06:33 James_F: Zuul: [mediawiki/extensions/PageImages] Add Scribunto phan dependency, for [[phab:T131911|T131911]]
* 06:33 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Add CLDR dependency

== 2025-05-03 ==
* 10:28 James_F: Zuul: [mediawiki/extensions/PageAssessments] Add Scribunto phan dependency, for [[phab:T380122|T380122]]

== 2025-05-02 ==
* 17:39 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Add Echo as a phan dep
* 12:30 James_F: Zuul: [mediawiki/extensions/CodeEditor] Add BetaFeatures phan dependency, for [[phab:T373711|T373711]]
* 12:18 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst voting again
* 08:43 hashar: Updating Quibble jobs to 1.14.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1140215 {{!}} [[phab:T378797|T378797]] [[phab:T384927|T384927]] [[phab:T386691|T386691]]
* 07:00 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Add cldr as full CI dep too, for [[phab:T391230|T391230]]
* 06:52 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Add cldr as phan dependency, for [[phab:T391230|T391230]]

== 2025-04-30 ==
* 23:46 dancy: Re-enabled https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/
* 18:54 dancy: Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad while Gerrit is down.
* 15:50 hashar: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1140203
* 15:01 hashar: Tagged Quibble 1.14.0 @ {{Gerrit|6d7c736d12daa7ea23b261ede02093f8fe7a83ae}} # [[phab:T378797|T378797]] [[phab:T384927|T384927]] [[phab:T386691|T386691]]
* 06:30 hashar: integration: cleared /srv/jenkins/workspace on integration-agent-docker-1062

== 2025-04-29 ==
* 21:04 mutante: integration-agent-docker-1051.integration - killall -9 ffmpeg - [[phab:T392963|T392963]]
* 20:28 mutante: integration-agent-docker-1048.integration - killall -9 ffpmeg - [[phab:T392963|T392963]]

== 2025-04-28 ==
* 19:01 taavi: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1139536
* 15:49 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/76
* 13:05 James_F: Docker: Bump Node20 and Node22 binaries to latest and cascade

== 2025-04-26 ==
* 00:05 bd808: Punched a hole in the beta cluster network blocks to allow 38.242.176.0/22 through.

== 2025-04-24 ==
* 19:54 thcipriani: deployment-cache-text08: systemctl reload varnish-frontend following puppet run change to /etc/varnish/blocked-nets.inc.vcl
* 19:49 thcipriani: deployment-cache-text08: sudo puppet-run to pick up https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/42c7880be27913c9e841642d9ff3e50deb455e08
* 15:32 bd808: Punched a hole in the beta cluster network blocks to allow 47.144.0.0/12 through. ([[phab:T392534|T392534]])
* 14:41 dancy: Updating runners to v17.9.3 in gitlab-cloud-runners (production)
* 14:34 dancy: Updating runners to v17.9.3 in gitlab-cloud-runners (staging)

== 2025-04-23 ==
* 22:59 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 22:43 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 22:15 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up a huge pile of new blocks ([[phab:T392534|T392534]])
* 22:11 James_F: Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on, for [[phab:T382177|T382177]]
* 21:47 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 21:29 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 20:47 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392534|T392534]])
* 17:43 James_F: Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now, for [[phab:T382177|T382177]]
* 16:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/a80e5211100f1cc42e4ae020d4266ea22938eb5a ([[phab:T383097|T383097]])
* 14:25 James_F: Zuul: [wikimedia/portals] Switch to Node 20, for [[phab:T382179|T382179]]

== 2025-04-17 ==
* 10:15 hashar: gerrit: reparented apps.git to All-Archived-Projects.git in order to BLOCK `mediawiki-replication`.  I have also archived all subprojects # [[phab:T392198|T392198]]

== 2025-04-16 ==
* 20:59 bd808: Blocked 193.43.72.0/24 and 14.160.0.0/11 because beta was very, very sad
* 16:02 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst non-voting for now
* 09:20 hashar: integration: restarted integration-puppetserver-01

== 2025-04-15 ==
* 22:02 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting, for [[phab:T368002|T368002]]
* 19:40 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks ([[phab:T392003|T392003]])
* 18:11 bd808: `bd808@deployment-cache-text08:~$ sudo service varnish-frontend restart` ([[phab:T392003|T392003]])
* 18:06 bd808: `sudo puppet agent -tv` on deployment-cache-text08 to update varnish deny list ([[phab:T392003|T392003]])
* 17:30 bd808: `shutdown -r now` on deployment-mediawiki14. Load has been growing for ~2 days.

== 2025-04-11 ==
* 19:53 James_F: Zuul: [oojs/router] Mark as archived, for [[phab:T391709|T391709]]
* 14:00 hashar: restarted integration-puppetserver: jvm went out of memory

== 2025-04-10 ==
* 23:40 bd808: Removed wikifunctions from deployment-cache prefix puppet's profile::cache::haproxy::available_unified_certificates::server_names. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/6af09ceaa6d261c910fb4b42d7b3e8b8172c8041%5E%21/
* 23:36 bd808: Deleted m.wikifunctions.beta.wmflabs.org, *.wikifunctions.beta.wmflabs.org, and wikifunctions.beta.wmflabs.org DNS records per [[Special:Diff/2292116]]. All 3 were pointing to 185.15.56.36.
* 14:16 hashar: deployment-prep: `profile::mediawiki::php::increase_open_files: True` on https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-mediawiki # [[phab:T389422|T389422]]
* 14:03 James_F: [Beta Cluster] On deployment-deploy04, running DELETE FROM localuser WHERE lu_wiki='wikifunctionswiki'; and DELETE FROM localnames WHERE ln_wiki='wikifunctionswiki'; for [[phab:T391511|T391511]]

== 2025-04-08 ==
* 22:39 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1135128
* 22:15 bd808: Manually deleted 'deployment-wikikube-v127' Magnum cluster template via Horizon. Deletion via OpenTofu has timed out repeatedly.
* 22:08 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1135123
* 22:02 brennen: Updating docker-pkg files on contint primary for [[phab:T383065|T383065]]
* 21:11 James_F: Beta Cluster: Shutting of deployment-docker-wikifunctions01, we decom'ing it.
* 20:44 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1135098

== 2025-04-07 ==
* 17:20 bd808: `service navtiming stop` to halt &quot;Unhandled exception in main loop, restarting consumer&quot; crash loop ([[phab:T391272|T391272]])
* 17:15 bd808: Reboot deployment-webperf21 ([[phab:T391272|T391272]])
* 16:58 bd808: `puppet agent -tv` to catch up with missed puppet runs on deployment-webperf21 ([[phab:T391272|T391272]])
* 16:56 bd808: `rm /var/log/user.log.1` on deployment-webperf21 ([[phab:T391272|T391272]])
* 16:47 bd808: `sudo /usr/local/sbin/clean-stale-puppet-certs --clean` on deployment-puppetserver-1 to clean up dangling certs for deployment-elastic&lt;nowiki&gt;{&lt;/nowiki&gt;09,10,11&lt;nowiki&gt;}&lt;/nowiki&gt;

== 2025-04-04 ==
* 09:42 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 35782 and 35784
* 09:09 hashar: Update tox jobs to default to python 3.9 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1134168
* 08:53 hashar: Updating Quibble jobs to catch up with latest image https://gerrit.wikimedia.org/r/c/integration/config/+/1134167 {{!}} [[phab:T3666646|T3666646]]
* 00:35 thcipriani: integration-agent-docker-1041 marked offline due to /srv disk space
* 00:09 Krinkle: Disable duplicate publishing noise from extension-MediaUploader, ref [[phab:T143162|T143162]], [[phab:T389450|T389450]]

== 2025-04-03 ==
* 15:06 James_F: Zuul: Configure the REL1_44 test and gate pipelines, for [[phab:T390695|T390695]]
* 13:33 James_F: Docker: [quibble-bullseye] Revert MardiaDB to 10.5, for (against) [[phab:T366646|T366646]]
* 13:08 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Publish JS docs

== 2025-04-02 ==
* 13:39 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133383 [[phab:T390754|T390754]]
* 12:36 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133379 https://gerrit.wikimedia.org/r/1133380
* 12:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133373

== 2025-04-01 ==
* 20:46 James_F: Zuul: Swap the branch check to specific release branches, for [[phab:T390754|T390754]] etc.
* 20:34 James_F: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, for [[phab:T366646|T366646]]
* 20:26 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133238
* 20:09 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133231
* 19:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133221 [[phab:T390754|T390754]]
* 18:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133209 [[phab:T390772|T390772]]
* 16:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1133184 [[phab:T390754|T390754]]

== 2025-03-31 ==
* 18:26 dancy: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1132688
* 15:20 James_F: Zuul: [mediawiki/extensions/EmailAuth] Mark as in Wikimedia production, move up, for [[phab:T390437|T390437]]
* 11:08 dcausse: [[phab:T389971|T389971]]: deleting deployment-elastic* VMs in deployment-prep
* 08:24 dcausse: [[phab:T389971|T389971]]: shutting down deployment-elastic* VMs in deployment-prep

== 2025-03-28 ==
* 22:01 Krinkle: Disable duplicate publishing noise from extension-LoginNotify, ref [[phab:T143162|T143162]], [[phab:T390315|T390315]]
* 21:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1130957
* 21:15 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1130957

== 2025-03-27 ==
* 16:28 bd808: Moved Puppet configuration from deployment-cache-text08 to deployment-cache-text prefix Puppet
* 16:05 bd808: `sudo systemctl restart varnish-frontend` on deployment-cache-text08 ([[phab:T390209|T390209]])
* 15:05 bd808: Moved role::acme_chief::cloud from individual instance config to deployment-acme-chief Puppet prefix.
* 00:55 bd808: Removed prefix puppet classes for deployment-acme-chief ([[phab:T390128|T390128]])

== 2025-03-26 ==
* 20:23 inflatador: bking@deployment-prep populating new OpenSearch cluster indices a la https://wikitech.wikimedia.org/w/index.php?title=Search&amp;oldid=2164435#Adding_new_wikis [[phab:T389971|T389971]]
* 17:10 inflatador: bking@deployment-prep reverted an accident replacement of deployment-acme-chief.yaml [[phab:T389971|T389971]]
* 15:04 dancy: Update gitlab-runners to v17.8.4 in gitlab-cloud-runners staging and production.
* 00:30 bd808: Delete parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud service name again ([[phab:T389252|T389252]])

== 2025-03-25 ==
* 21:11 jeena: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1130722
* 04:18 jeena: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1130729

== 2025-03-24 ==
* 19:35 hashar: Updating Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1130700

== 2025-03-23 ==
* 18:41 James_F: Zuul: Add 0xDeadbeef to CI allowlist
* 18:34 James_F: Zuul: [operations/debs/bdsync] Mark as archived, for [[phab:T377882|T377882]]
* 18:31 James_F: Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency, for [[phab:T386435|T386435]]
* 18:29 James_F: Zuul: [mediawiki/extensions/CategoryWatch] Add Echo CI dependency

== 2025-03-20 ==
* 23:31 bd808: integration: thcipriani added integration-agent-docker-106&lt;nowiki&gt;{&lt;/nowiki&gt;0,1,2&lt;nowiki&gt;}&lt;/nowiki&gt; earlier today ([[phab:T389554|T389554]])
* 22:50 brennen: integration: added jenkins nodes for integration-agent-docker-106&lt;nowiki&gt;{&lt;/nowiki&gt;3,4,5&lt;nowiki&gt;}&lt;/nowiki&gt; with 3 executors per each ([[phab:T389554|T389554]])
* 21:41 brennen: integration: launched integration-agent-docker-106&lt;nowiki&gt;{&lt;/nowiki&gt;3,4,5&lt;nowiki&gt;}&lt;/nowiki&gt; ([[phab:T389554|T389554]])
* 21:25 eileen: civicrm upgraded from {{Gerrit|7b532ad7}} to {{Gerrit|fba4c3d6}}
* 15:13 dancy: Rebooting integration-agent-docker-1046 (Seems to be be inaccessible since February)
* 08:28 taavi: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1129765

== 2025-03-19 ==
* 20:32 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1129364
* 00:12 bd808: Trying the simplest thing that might work by adding a CNAME record for parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud. ([[phab:T389252|T389252]])

== 2025-03-18 ==
* 20:25 bd808: Rebooting deployment-jobrunner05 because things just seem weird ([[phab:T387631|T387631]], [[phab:T387276|T387276]])
* 15:18 sergi0: run CommunityUpdates config schema migration `foreachwikiindblist growthexperiments extensions/CommunityConfiguration/maintenance/migrateConfig.php CommunityUpdates` ([[phab:T387737|T387737]])

== 2025-03-14 ==
* 21:36 Reedy: deployed https://gerrit.wikimedia.org/r/1127982
* 16:55 Lucas_WMDE: manually killed job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81/2928/console which had been stuck since 16:33 UTC, blocking gate-and-submit :(

== 2025-03-13 ==
* 21:29 dancy: Finished gitlab cloud runners k8s production cluster upgrade ([[phab:T388836|T388836]])
* 20:42 dancy: Finished gitlab cloud runners k8s staging cluster upgrade ([[phab:T388836|T388836]])
* 20:09 dancy: Starting gitlab cloud runners k8s production cluster upgrade ([[phab:T388836|T388836]])
* 19:26 dancy: Starting gitlab cloud runners k8s staging cluster upgrade ([[phab:T388836|T388836]])

== 2025-03-11 ==
* 22:54 bd808: Deleted unattached volumes: alert01, db09, deploy03, mwmaint, ores02, parsoid14-srv, prometheus05
* 22:39 bd808: Released unused floating IPs 185.15.56.9 and 185.15.56.97 back to global pool
* 22:08 bd808: Updated mail.beta.wmflabs.org service name to point to 185.15.56.115
* 22:04 bd808: Deleted orphan parsoid-external-ci-access.beta.wmflabs.org. DNS record
* 21:53 bd808: Deleted dangling prometheus-beta.wmcloud.org web proxy
* 21:50 bd808: Deleted dangling w-beta.wmflabs.org web proxy
* 21:42 bd808: Deleted unused &quot;deployment-parsoid&quot; Prefix Puppet configuration
* 20:48 James_F: Docker: [quibble-bullseye-php81 &amp; php81] Use PCRE2 backport from component/php81, for [[phab:T386006|T386006]]
* 13:19 James_F: Zuul: [mediawiki/extensions/ActiveAbstract] Mark as archived, for [[phab:T382069|T382069]]
* 03:54 eileen: civicrm upgraded from {{Gerrit|f2222fcd}} to {{Gerrit|ec20a105}}

== 2025-03-10 ==
* 15:20 James_F: Zuul: [mediawiki/services/servicelib-node] Mark as archived, for [[phab:T388424|T388424]]
* 13:47 hashar: gerrit: removed leftover empty directory `/srv/gerrit/plugins/lfs`. Data have been migrated to `/srv/gerrit/plugins/lfs` as part of moving Gerrit data out of `/`. See [[phab:T333143|T333143]]

== 2025-03-08 ==
* 01:22 James_F: Zuul: [php-session-serializer] Enable PHP 8.4 as voting, for [[phab:T368270|T368270]]

== 2025-03-07 ==
* 21:00 James_F: Zuul: [mediawiki/libs/Shellbox] Enable PHP 8.4 as voting, for [[phab:T386570|T386570]]
* 20:53 James_F: Zuul: [wikipeg] Enable PHP 8.4 as voting, for [[phab:T386570|T386570]]
* 20:07 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.4 as voting, for [[phab:T387806|T387806]]

== 2025-03-05 ==
* 00:21 dancy: Reeanbled beta-scap-sync-world  ([[phab:T166010|T166010]])

== 2025-03-04 ==
* 23:26 dancy: Disabling beta-scap-sync-world for noise reduction while dealing with [[phab:T166010|T166010]]
* 22:10 James_F: Zuul: [mediawiki/services/example-node-api] Mark as archived, for [[phab:T387933|T387933]]
* 01:42 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4, for [[phab:T386570|T386570]]
* 01:13 James_F: Zuul: Add WgevaertWikiBase to CI allowlist
* 01:03 James_F: Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos, for [[phab:T386108|T386108]]

== 2025-02-28 ==
* 18:20 dancy: Upgrading gitlab-runner to v17.7.1 in production gitlab-cloud-runners ([[phab:T386297|T386297]])
* 18:12 dancy: Upgrading gitlab-runner to v17.7.1 in staging gitlab-cloud-runners ([[phab:T386297|T386297]])
* 17:52 dancy: Upgraded scap to 4.138.0 in beta cluster
* 16:43 bd808: Deleted now dangling parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud. DNS record ([[phab:T385849|T385849]])
* 16:40 bd808: Deleted deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T385849|T385849]])
* 16:39 bd808: Deleted parsoid-external-ci-access.wmcloud.org proxy ([[phab:T385849|T385849]])
* 16:37 bd808: Deleted deployment-alert01.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T385849|T385849]])
* 16:36 bd808: Deleted deployment-bastion.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T385849|T385849]])

== 2025-02-27 ==
* 01:11 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1123063 [[phab:T386476|T386476]]

== 2025-02-26 ==
* 20:21 James_F: jforrester@doc1003:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/LdapAuthentication/ #[[phab:T376097|T376097]]
* 20:18 James_F: Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived, for [[phab:T376097|T376097]]
* 13:20 hashar: Updating Quibble jobs to 1.13.0.  &quot;Skip execution upon a success cache hit&quot; which would make some jobs to skip tests entirely when a set of commits/image is known to have previously passed # [[phab:T383243|T383243]] {{!}} dduvall
* 11:06 hashar: Tag Quibble 1.13.0 @ {{Gerrit|0ac128f7bc060c82f11317aabaf78a10b24aeeec}} # [[phab:T383243|T383243]]
* 09:11 hashar: deployment-prep: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1122901 &quot;php: use component/pcre2 when using Php 8.1&quot; to fix php # [[phab:T387276|T387276]]
* 01:55 bd808: `./jjb-update 'integration-quibble-fullrun-*-php81' '*-php81-phan' '*php81*'`
* 01:16 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1122700 [[phab:T386006|T386006]]

== 2025-02-25 ==
* 20:25 James_F: Docker: [php81] Update PHP to 8.1.31-1+wmf11u4, for [[phab:T386006|T386006]]
* 14:07 James_F: Docker: [php81] Upgrade Wikimedia's PHP to 8.1.31-1+wmf11u3 &amp; PCRE to 10.42 for [[phab:T386006|T386006]]

== 2025-02-24 ==
* 01:02 jeena: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/73

== 2025-02-22 ==
* 11:27 taavi: rebooting integration-agent-docker-1047 which thinks it is gerrit

== 2025-02-21 ==
* 22:54 brennen: gitlab: removing expiration date for 14 tokens expiring in 2025 ([[phab:T385930|T385930]])
* 22:36 brennen: gitlab: set require_personal_access_token_expiry and service_access_tokens_expiration_enforced to false

== 2025-02-20 ==
* 20:15 dancy: Updated buildkitd to v0.20.0 in gitlab-cloud-runners ([[phab:T386955|T386955]])
* 20:15 dancy: Updated buildkitd to v0.20.0 in gitlab-cloud-runners

== 2025-02-19 ==
* 21:28 dancy: Reenabled https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/ ([[phab:T386851|T386851]])
* 19:35 dduvall: restarting jenkins to fix git related issues following java update ([[phab:T386755|T386755]])
* 15:47 dancy: Disabled the https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ job to reduce noise while the problem is being debugged.

== 2025-02-18 ==
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1119815
* 16:11 James_F: Zuul: [operations/debs/dnsdist] Revert archival

== 2025-02-13 ==
* 13:57 James_F: Zuul: [mediawiki/extensions/CirrusSearch] Drop WikibaseCirrusSearch dep, for [[phab:T386015|T386015]]

== 2025-02-12 ==
* 17:22 James_F: Zuul: Add User:Michi j to CI allowlist
* 17:21 James_F: Zuul: Add Dragoniez to CI allowlist

== 2025-02-11 ==
* 15:43 James_F: Zuul: Make PHP 8.4 voting on lib repos where it already passes, for [[phab:T386108|T386108]]

== 2025-02-10 ==
* 14:27 James_F: Zuul: Add Bunnypranav to CI allowlist

== 2025-02-08 ==
* 00:07 bd808: Added `profile::maps::osm_master::disable_waterlines_import_timer: false` to deployment-maps prefix hiera ([[phab:T385921|T385921]])

== 2025-02-07 ==
* 22:14 brennen: phab/phorge: replaced mr-widget token in deployed config ([[phab:T385480|T385480]])
* 21:33 bd808: Added `profile::restbase::parsoid_uri: https://phabricator.wikimedia.org/T385902` to deployment-restbase prefix puppet ([[phab:T385902|T385902]])
* 01:34 bd808: Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117997 to deployment-puppetmaster ([[phab:T385849|T385849]])
* 00:42 bd808: Shutoff deployment-parsoid14 to see if anything breaks/anyone yells ([[phab:T385849|T385849]])

== 2025-02-06 ==
* 23:53 bd808: Updated citoid-beta.wmflabs.org to point to deployment-docker-citoid02
* 23:50 bd808: Deleted beta-prometheus.wmflabs.org; it was pointed to an IP now owned by the mdwikioffline project.
* 23:43 bd808: Deleted recently orphaned spiderpig.wmcloud.org proxy after discussion with dancy
* 16:20 bd808: Rebooted deployment-sessionstore06 ([[phab:T385803|T385803]])
* 12:07 andrewbogott: rebooting all servers for [[phab:T385264|T385264]]

== 2025-02-05 ==
* 19:17 James_F: Zuul: [mediawiki/extensions/DonationInterface] Switch CI from PHP74 to PHP82
* 18:23 James_F: Zuul: [mediawiki/extensions/cldr] Raise FR-special job to REL1_43
* 18:22 James_F: Zuul: [mediawiki/extensions/DonationInterface] Raise FR-special job to REL1_43
* 18:11 James_F: Zuul: [labs/tools/heritage] Fold template into this, only user
* 18:08 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Test in PHP 8.2+ only
* 17:29 James_F: Zuul: [mediawiki/core] Test fundraising branches against PHP 8.2
* 17:19 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Mark as non-prod

== 2025-02-03 ==
* 12:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115782

== 2025-01-30 ==
* 15:12 James_F: Zuul: [mediawiki/extensions/Wikibase] Only inject EntitySchema on 1.43+, for [[phab:T385175|T385175]]
* 01:39 James_F: Zuul: [mediawiki/core] Remove composer variant from wmf branches
* 00:42 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115131

== 2025-01-29 ==
* 18:03 James_F: Zuul: Make FR REL1_43-php82 voting for cldr and FEU
* 17:54 James_F: Zuul: Add FR REL1_43-php82 as experimental to other extensions
* 17:40 James_F: Zuul: [mediawiki/extensions/cldr] Add FR REL1_43-php82 as experimental
* 17:40 James_F: Zuul: [mediawiki/extensions/cldr] Re-enable FR-tech job as voting, passes fine
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115064
* 16:33 hashar: gerrit: marked all legacy Puppet modules as read-only ( https://gerrit.wikimedia.org/r/admin/repos/q/filter:operations/puppet/ ) and removed the associated GitHub mirrors that existed for some of them

== 2025-01-28 ==
* 17:46 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1113550 ([[phab:T383337|T383337]])
* 17:38 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1113549 ([[phab:T383337|T383337]])
* 10:07 hashar: Manually cleaned integration-agent-docker-1043

== 2025-01-27 ==
* 18:17 hashar: Cleaned disk on integration-agent-docker-1051

== 2025-01-25 ==
* 09:20 taavi: reloading zuul for https://gerrit.wikimedia.org/r/1113739

== 2025-01-24 ==
* 21:44 James_F: Revert &quot;Zuul: Switch Fundraising jobs to REL1_43&quot;

== 2025-01-23 ==
* 16:31 dancy: Updating production gitlab-cloud-runners to v17.6.1
* 16:23 dancy: Updating staging gitlab-cloud-runners to v17.6.1

== 2025-01-22 ==
* 18:14 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add Wikibase as a phan dependency

== 2025-01-20 ==
* 09:55 hashar: Updating Quibble jobs to enable success cache experiment - [[phab:T383243|T383243]]
* 08:20 hashar: Updating all Jenkins jobs to update Quibble to 1.12.0

== 2025-01-17 ==
* 16:59 dduvall: Building Docker images for Quibble 1.12.0
* 15:00 hashar: Building Docker images for Quibble 1.12.0
* 12:56 hashar: Tag Quibble 1.12.0 @ {{Gerrit|633099ead3ec72180e7890e1980074b4fde56c26}} # [[phab:T365978|T365978]], [[phab:T383243|T383243]]

== 2025-01-14 ==
* 17:14 brennen: integration project: create integration-agent-docker-1059 for [[phab:T383254|T383254]]
* 16:50 brennen: integration project: create integration-agent-docker-1058 for [[phab:T383254|T383254]]

== 2025-01-10 ==
* 15:55 dancy: Updating gitlab-cloud-runners (prod) to v17.5.5 ([[phab:T383263|T383263]])
* 15:49 dancy: Updating gitlab-cloud-runners (staging) to v17.5.5

== 2025-01-09 ==
* 22:20 brennen: gitlab: Feature.enable(:kubernetes_agent_protected_branches) - https://docs.gitlab.com/ee/user/clusters/agent/ci_cd_workflow.html#restrict-access-to-the-agent-to-protected-branches
* 18:08 James_F: Docker: [node22] Update Node to v22.13.0, &amp; switch base image to bookworm, for [[phab:T383337|T383337]]
* 17:01 James_F: Docker: [node20] Update Node to v20.18.1, &amp; switch base image to bookworm, for [[phab:T383337|T383337]]
* 15:13 James_F: Docker: [sury-php] Re-platform to bookworm

== 2025-01-08 ==
* 22:07 hashar: castor: deleting potentially corrupted npm cache. On integration-castor05: sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/&lt;nowiki&gt;{&lt;/nowiki&gt;wmf-quibble-selenium-php74,quibble-vendor-mysql-php74-selenium&lt;nowiki&gt;}&lt;/nowiki&gt;/npm  # [[phab:T383237|T383237]]

== 2025-01-07 ==
* 22:07 hashar: Deleted /srv/zuul/git/operations/dumps/dcat on both contint1002 and contint2002 # [[phab:T157818|T157818]]
* 19:00 bd808: `/usr/local/sbin/clean-stale-puppet-certs --clean` ([[phab:T383153|T383153]])
* 18:53 taavi: taavi@deployment-puppetserver-1:~$ sudo puppetserver ca clean --certname maps-master01.maps-experiments.eqiad1.wikimedia.cloud # [[phab:T383153|T383153]]
* 18:50 taavi: taavi@deployment-puppetserver-1:~$ sudo puppet node clean geoshapes.maps-experiments.eqiad1.wikimedia.cloud # [[phab:T383153|T383153]]
* 18:30 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance deployment-etcd04
* 18:30 bd808@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance deployment-etcd04
* 14:48 hashar: Manually renamed wikibase-daily-npm-audit-daily-node18-npmaudit to node20 variant and refresh the config with JJB
* 14:33 James_F: Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master

== 2025-01-06 ==
* 20:16 andrewbogott: removed the (non-existent?) role::mw_rc_irc from puppet config for deployment-ircd03.deployment-prep.eqiad1.wikimedia.cloud
* 19:35 bd808: Manually generated missing en_US.UTF-8 locale on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T361381|T361381]])
* 19:32 bd808: Added `postgresql::postgis::postgresql_postgis_package: postgresql-15-postgis-3` to deployment-maps Prefix Puppet to work around default parameter problem ([[phab:T361381|T361381]])
* 19:31 bd808: Issued new Puppet cert for deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud ([[phab:T361381|T361381]])
* 19:27 bd808: Added `postgresql::postgis::postgresql_postgis_package: ignored` to deployment-maps Prefix Puppet to work around default parameter problem ([[phab:T361381|T361381]])
* 19:15 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 ([[phab:T382709|T382709]])
* 19:11 bd808: Added placeholders for `graphite_host` and `statsd` to deployment-webperf Prefix Puppet
* 18:53 bd808: Fixed missing profile::swift::global_account_keys::&lt;nowiki&gt;{&lt;/nowiki&gt;codfw, eqiad&lt;nowiki&gt;}&lt;/nowiki&gt; placeholders breaking deployment-ms-* puppet runs
* 18:38 bd808: Fixed incorrect deployment-restbase prefix puppet setting that was causing puppet run failures
* 18:19 bd808: Issued a new Puppet client cert for traindev01.deployment-prep.eqiad1.wikimedia.cloud
* 14:58 James_F: Zuul: Drop CI for REL1_41 branch, now EOL per [[phab:T376550|T376550]]
* 09:03 hashar: gerrit: flushed diff_intraline, diff_summary, gerrit_file_diff and git_file_diff caches after having turned on diff3 style # [[phab:T359821|T359821]]

== 2025-01-02 ==
* 11:27 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1105679 # [[phab:T374113|T374113]]


{{SAL-archives/Release Engineering}}

&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>lhwg04gx8qy78wjye4z8ybpieop8pga</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.quickcategories/SAL</title>
    <ns>498</ns>
    <id>443800</id>
    <revision>
      <id>2320873</id>
      <parentid>2314860</parentid>
      <timestamp>2025-07-07T07:03:15Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~lucaswerkmeister@tools-bastion-13: deployed 5b65c99d03 (upgrade dependencies, including mwparserfromhell 0.7.2)</comment>
      <origin>2320873</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="25868" sha1="ooflckj507pk6nm97lyecuqfjh9ipeb" xml:space="preserve">=== 2025-07-07 ===
* 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2)

=== 2025-06-18 ===
* 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&amp;actions= and &amp;title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]])

=== 2025-06-11 ===
* 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]])

=== 2025-05-22 ===
* 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]])
* 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies)

=== 2025-05-12 ===
* 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]])

=== 2025-02-24 ===
* 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR &amp;&amp; webservice restart

=== 2025-02-20 ===
* 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to &lt;a href=&quot;https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/&quot;&gt;database maintenance&lt;/a&gt;.' &amp;&amp; webservice restart

=== 2025-02-03 ===
* 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to &lt;a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/&gt;necessary database maintenance&lt;/a&gt;.' &amp;&amp; webservice restart

=== 2024-12-16 ===
* 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request])

=== 2024-12-03 ===
* 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index)

=== 2024-12-01 ===
* 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior)

=== 2024-11-29 ===
* 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column)

=== 2024-11-24 ===
* 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1)

=== 2024-10-25 ===
* 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6)

=== 2024-10-13 ===
* 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^

=== 2024-10-11 ===
* 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]])

=== 2024-10-10 ===
* 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better)

=== 2024-09-28 ===
* 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies)
* 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift)
* 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing)
* 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing)
* 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op)

=== 2024-09-18 ===
* 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]])
* 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/
* 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' &gt; service.template
* 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner &amp;&amp; toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous
* 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop &amp;&amp; webservice --mount=none buildservice start
* 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories

=== 2024-09-17 ===
* 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config)
* 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner &amp;&amp; kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk
* 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars)
* 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z
* 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL &quot;'https://editgroups-commons.toolforge.org/b/QC/&lt;nowiki&gt;{&lt;/nowiki&gt;0&lt;nowiki&gt;}&lt;/nowiki&gt;/'&quot;
* 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org

=== 2024-09-16 ===
* 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead
* 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories
* 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true

=== 2024-09-15 ===
* 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]])
* 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead
* 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;consumer_secret&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET
* 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;consumer_key&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY
* 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead
* 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;SUMMARY_BATCH_LINK&quot;]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK
* 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago

=== 2024-09-14 ===
* 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead
* 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY &quot;$(python3 -c 'import yaml; print(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;SECRET_KEY&quot;])')&quot;
* 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet)
* 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config)
* 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now

=== 2024-08-09 ===
* 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck?

=== 2024-06-12 ===
* 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]]

=== 2024-04-08 ===
* 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path)
* 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login)

=== 2024-01-04 ===
* 12:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6)

=== 2023-12-28 ===
* 13:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup)
* 11:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions)

=== 2023-10-25 ===
* 18:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1)

=== 2023-10-03 ===
* 12:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop)
* 11:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3)

=== 2023-09-07 ===
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major)

=== 2023-07-15 ===
* 13:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions)
* 13:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7fd188d540}} (Python 3.11)

=== 2023-05-01 ===
* 23:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq)

=== 2023-04-29 ===
* 15:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3)

=== 2023-04-06 ===
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check)
* 16:17 wm-bot: &lt;lucaswerkmeister&gt; set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance

=== 2023-04-05 ===
* 19:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references)

=== 2023-04-01 ===
* 14:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls)

=== 2023-03-28 ===
* 19:11 wm-bot: &lt;lucaswerkmeister&gt; unset EXPECTED_DATABASE_ERROR again

=== 2023-03-27 ===
* 16:39 wm-bot: &lt;lucaswerkmeister&gt; set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance

=== 2023-03-22 ===
* 23:41 wm-bot: &lt;lucaswerkmeister&gt; unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future

=== 2023-02-21 ===
* 00:43 wm-bot: &lt;lucaswerkmeister&gt; optimized querytime table (~tools.quickcategories/purge-querytime)
* 00:43 wm-bot: &lt;lucaswerkmeister&gt; END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime)

=== 2023-02-20 ===
* 23:29 wm-bot: &lt;lucaswerkmeister&gt; START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime)
* 23:28 wm-bot: &lt;lucaswerkmeister&gt; optimized querytime table (~tools.quickcategories/purge-querytime)
* 23:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain)
* 10:35 wm-bot: &lt;lucaswerkmeister&gt; set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime

=== 2023-02-14 ===
* 21:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬)
* 20:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes)

=== 2022-09-10 ===
* 18:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart)
* 18:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|306045651d}} (diffusion → gitlab)

=== 2022-06-24 ===
* 15:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart)

=== 2022-04-19 ===
* 16:29 wm-bot: &lt;lucaswerkmeister&gt; removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over

=== 2022-04-13 ===
* 20:18 wm-bot: &lt;lucaswerkmeister&gt; optimized querytime table (~tools.quickcategories/purge-querytime)
* 20:18 wm-bot: &lt;lucaswerkmeister&gt; END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime)
* 19:04 wm-bot: &lt;lucaswerkmeister&gt; START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime)
* 18:30 wm-bot: &lt;lucaswerkmeister&gt; added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage

=== 2022-03-26 ===
* 17:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op
* 17:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45239001ac}} (pip-tools)

=== 2022-02-15 ===
* 19:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3627eb9988}} (fix query)
* 19:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bb178f29b2}} (missing template files)
* 19:05 wm-bot: &lt;lucaswerkmeister&gt; restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d)
* 19:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now
* 18:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner
* 18:21 wm-bot: &lt;lucaswerkmeister&gt; deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it
* 18:18 wm-bot: &lt;lucaswerkmeister&gt; updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps)

=== 2021-10-13 ===
* 18:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2

=== 2021-09-25 ===
* 14:48 wm-bot: &lt;lucaswerkmeister&gt; removed old venv-3.7

=== 2021-09-18 ===
* 13:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc)

=== 2021-09-14 ===
* 22:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e8a95d4c04}} (background runner speedup)
* 22:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv)
* 20:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups)

=== 2021-09-03 ===
* 16:05 wm-bot: &lt;lucaswerkmeister&gt; updated venv (includes mwparserfromhell 0.6.3)

=== 2021-08-08 ===
* 22:08 wm-bot: &lt;lucaswerkmeister&gt; END - optimize querytime table
* 22:08 wm-bot: &lt;lucaswerkmeister&gt; START - optimize querytime table
* 22:02 wm-bot: &lt;lucaswerkmeister&gt; END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows)
* 13:16 wm-bot: &lt;lucaswerkmeister&gt; START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches

=== 2021-08-07 ===
* 20:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7028c292e7}} (lock less tables)
* 19:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bbdebedd08}} (styling improvement)

=== 2021-07-20 ===
* 19:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1d17bc8e87}} (update config loading code)
* 19:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b1f23a7801}} (minor updates)

=== 2021-05-30 ===
* 15:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner)
* 15:14 wm-bot: &lt;lucaswerkmeister&gt; restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one)
* 15:12 wm-bot: &lt;lucaswerkmeister&gt; started webservice again (background runner not yet, odd kubectl error)
* 15:12 wm-bot: &lt;lucaswerkmeister&gt; updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1)
* 15:10 wm-bot: &lt;lucaswerkmeister&gt; (this includes the background runner as well)
* 15:09 wm-bot: &lt;lucaswerkmeister&gt; temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?)

=== 2021-04-09 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]])

=== 2021-04-07 ===
* 20:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]])

=== 2021-02-28 ===
* 19:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op)
* 19:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a01dae7728}} (better OAuth error handling)

=== 2021-02-26 ===
* 10:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index)

=== 2021-02-24 ===
* 21:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b2f83f22c0}} (a11y fix)

=== 2021-02-16 ===
* 20:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cc50c3ee7a}} (add skip link)
* 19:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1265f0d223}} (Bootstrap update)

=== 2020-12-28 ===
* 09:00 wm-bot: &lt;lucaswerkmeister&gt; updated Python packages

=== 2020-12-17 ===
* 17:24 wm-bot: &lt;lucaswerkmeister&gt; removed expected_database_error from config, ToolsDB maintenance is over

=== 2020-12-16 ===
* 16:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance)

=== 2020-12-14 ===
* 20:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8c9a2dfc59}} (fix current_url)

=== 2020-11-14 ===
* 22:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f10c62ced1}} (new export mode)

=== 2020-11-10 ===
* 15:57 wm-bot: &lt;lucaswerkmeister&gt; made tool readonly to avoid issues during ToolsDB maintenance

=== 2020-11-02 ===
* 16:17 wm-bot: &lt;lucaswerkmeister&gt; tool is read-write again, ToolsDB maintenance was rescheduled (again)
* 15:56 wm-bot: &lt;lucaswerkmeister&gt; made tool readonly to avoid issues during ToolsDB maintenance

=== 2020-10-27 ===
* 16:25 wm-bot: &lt;lucaswerkmeister&gt; tool is read-write again, ToolsDB maintenance was rescheduled
* 15:55 wm-bot: &lt;lucaswerkmeister&gt; made tool readonly to avoid issues during ToolsDB maintenance

=== 2020-10-17 ===
* 15:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart)

=== 2020-06-26 ===
* 21:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e1c86c5b27}} (update pagepile url)

=== 2020-06-15 ===
* 20:47 wm-bot: &lt;lucaswerkmeister&gt; renamed default branch from master to main

=== 2020-06-14 ===
* 11:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c0be64fafd}} (layout fix)

=== 2020-04-25 ===
* 17:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|90342028c5}} (toolforge.org)

=== 2020-02-20 ===
* 23:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes)

=== 2020-02-16 ===
* 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself)

=== 2020-02-02 ===
* 17:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month

=== 2019-11-08 ===
* 17:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning)

=== 2019-09-08 ===
* 20:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b99c878d2a}} (db schema cleanup)
* 19:35 wm-bot: &lt;lucaswerkmeister&gt; reset command 266888 (stuck in pending status) and restarted background runner

=== 2019-09-02 ===
* 20:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only)

=== 2019-08-31 ===
* 18:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF)

=== 2019-08-08 ===
* 17:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6b45be2605}} (fix wiki read-only display)

=== 2019-06-30 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b1a75f69dc}} (remove debug code)
* 22:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|20a47a955b}} (preferences linked in navbar)
* 18:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version)

=== 2019-06-19 ===
* 11:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a35b558374}} (add logout route)

=== 2019-06-18 ===
* 20:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists)

=== 2019-06-16 ===
* 14:37 wm-bot: &lt;lucaswerkmeister&gt; deployed temporary change (hide notifications count for Harmonia Amanda per request)
* 14:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a415a923dd}} (show notifications count)

=== 2019-06-01 ===
* 20:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|32b8528d58}} (export options, including PagePile export)
* 12:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7ba32d514c}} (PagePile support)

=== 2019-05-31 ===
* 22:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ebfed74416}} (minor error handling improvements)
* 16:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c58cc0810}} (minor UI improvements)

=== 2019-05-30 ===
* 22:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder)
* 22:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix)
* 22:12 wm-bot: &lt;lucaswerkmeister&gt; git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]]

=== 2019-05-18 ===
* 21:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility)
* 20:52 wm-bot: &lt;lucaswerkmeister&gt; rebuilt webservice venv for Python 3.5
* 20:46 wm-bot: &lt;lucaswerkmeister&gt; switch webservice to python3.5
* 20:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a0a6530cfe}} (rearrange buttons)
* 09:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|83d01bb93b}} (background UI improvements)

=== 2019-05-17 ===
* 21:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57faa03444}} (fix tabindex)

=== 2019-05-16 ===
* 18:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page)

=== 2019-05-15 ===
* 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]])

=== 2019-05-12 ===
* 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix)
* 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements)

=== 2019-05-11 ===
* 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles)
* 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times)

=== 2019-05-09 ===
* 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times)

=== 2019-05-04 ===
* 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements)
* 20:52 lucaswerkmeister: installed beautifulsoup4
* 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱)
* 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch)
* 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index)
* 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list)

=== 2019-05-03 ===
* 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period)
* 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary)

=== 2019-05-02 ===
* 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page)

=== 2019-05-01 ===
* 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries)
* 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles)
* 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}}
* 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned
* 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned
* 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned

=== 2019-04-30 ===
* 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}}

=== 2019-04-28 ===
* 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}}
* 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}}

=== 2019-04-22 ===
* 20:16 lucaswerkmeister: enable background runs for everyone
* 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}}

=== 2019-04-18 ===
* 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}}

=== 2019-04-14 ===
* 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ooflckj507pk6nm97lyecuqfjh9ipeb</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.lexeme-forms/SAL</title>
    <ns>498</ns>
    <id>443946</id>
    <revision>
      <id>2320872</id>
      <parentid>2313354</parentid>
      <timestamp>2025-07-07T06:33:59Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~lucaswerkmeister@tools-bastion-13: deployed cc28ff0494 (l10n updates: et, it, nn, pt-br, ru)</comment>
      <origin>2320872</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="70111" sha1="t97e9jap0spv4y5xxntl0b0olwffzdb" xml:space="preserve">=== 2025-07-07 ===
* 06:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cc28ff0494}} (l10n updates: et, it, nn, pt-br, ru)

=== 2025-06-16 ===
* 17:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c7b450ab94}} (update code for newer mwapi version)

=== 2025-06-11 ===
* 12:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cae8c3c341}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]])

=== 2025-05-31 ===
* 13:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|706110e863}} (l10n updates: da, lb)

=== 2025-05-13 ===
* 16:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8a583bf6ff}} (l10n updates: tg)

=== 2025-05-06 ===
* 17:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7349295f62}} (l10n updates: el)
* 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e969f66351}} (update absolute_construction item ID)

=== 2025-04-22 ===
* 23:10 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d9516b6b1c}} (Quechua verb Wikifunctions)

=== 2025-04-21 ===
* 18:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c6d552c9c3}} (upgrade dependencies, including toolforge-i18n 0.1.2)

=== 2025-04-19 ===
* 10:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5425b40c0f}} (l10n updates: es)

=== 2025-04-14 ===
* 19:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ae10863f8f}} (l10n updates: af)

=== 2025-04-07 ===
* 18:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e106b7b684}} (Quechua verbs + l10n updates: es, pa, qu, zh-hant)

=== 2025-04-04 ===
* 19:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a377a0be8c}} (remove unneeded CSS)

=== 2025-03-29 ===
* 21:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|98e408e5a6}} (Russian perfective verbs)

=== 2025-03-15 ===
* 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ab6621b22d}} (l10n updates: ar)

=== 2025-03-11 ===
* 20:35 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6def84813}} (l10n updates: lb)

=== 2025-02-21 ===
* 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|81611bc5dc}} (l10n updates: pa, tr)

=== 2025-02-04 ===
* 21:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2ccb28ad17}} (l10n updates: lb)

=== 2025-01-24 ===
* 10:21 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|223cafa209}} (l10n updates: ms)

=== 2025-01-09 ===
* 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cebad0e4dd}} (l10n updates: ia, pa)

=== 2025-01-06 ===
* 20:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e7e3f2a500}} (l10n updates: cs, he)

=== 2024-12-21 ===
* 22:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|eb9d0ae3c2}} (l10n updates: lb, pa; also upgrade dependencies, including Flask 3.1.0 and Jinja2 3.1.5)

=== 2024-12-12 ===
* 22:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5ffdfb2c55}} (l10n updates: he, nl)

=== 2024-11-18 ===
* 19:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3933dbfa7f}} (l10n updates: af, ar, de, fr, gl, he, krc, mk, pa, sk, zh-hans); manually restored sh-latn ([[phab:T379188|T379188]])

=== 2024-11-04 ===
* 17:32 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop; webservice start # [[phab:T378976|T378976]]

=== 2024-11-02 ===
* 16:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a6768b885c}} (add setting for using Wikifunctions)
* 14:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4fdd9491ee}} (improve Wikifunctions UI)
* 09:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f28c3414ad}} (upgrade dependencies, including Werkzeug 3.1.0); also upgraded pip from 24.2 to 24.3.1

=== 2024-10-25 ===
* 19:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8cdbda6ce3}} (upgrade dependencies, including Werkzeug 3.0.6)

=== 2024-10-13 ===
* 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bfcaca2fa3}} (upgrade dependencies, including MarkupSafe 3.0)

=== 2024-10-03 ===
* 16:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6377a9095}} (upgrade dependencies, including toolforge_i18n 0.1.1 and Werkzeug 3.0.4)
* 13:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a81b469204}} (l10n updates: ms-arab)

=== 2024-09-26 ===
* 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|96f45731db}} (l10n updates: ar, ms-arab)

=== 2024-09-11 ===
* 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|38b3b281ed}} (fix two ZIDs for Breton templates)

=== 2024-09-01 ===
* 14:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|53a1efcc14}} (l10n updates: cy, uk)

=== 2024-08-18 ===
* 12:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|309b33b80b}} (l10n updates: pl, tg)
* 12:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6deace1e36}} (Italian masculine+feminine nouns, dependency upgrades)

=== 2024-08-12 ===
* 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|246f9d26da}} (l10n updates: tg)

=== 2024-08-05 ===
* 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e3448958a0}} (upgrade toolforge_i18n to 0.0.7)

=== 2024-07-31 ===
* 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4775170045}} (upgrade toolforge_i18n to 0.0.6; also upgrade pip to 24.2)

=== 2024-07-26 ===
* 21:11 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bb61fc3c89}} (l10n updates: vi [no actual translation changes, one addition to the authors, presumably their edit got reverted])

=== 2024-07-22 ===
* 18:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13c4824e3a}} (change Babel code of kaa from kk to uz)

=== 2024-07-21 ===
* 18:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e856c9b2d2}} (upgrade toolforge_i18n to 0.0.5; also upgrade pip to 24.1.2)

=== 2024-07-08 ===
* 18:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6fa2d82b8}} (l10n updates: ja)

=== 2024-07-07 ===
* 18:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f3b3981ec9}} (upgrade toolforge_i18n to 0.0.2; also upgrade pip from 24.0 to 24.1.1)

=== 2024-07-05 ===
* 12:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1013a7234d}} (l10n updates: ar, de, uk)

=== 2024-06-18 ===
* 19:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8530f5f235}} (l10n updates: eo, fa, kaa, lb)

=== 2024-06-15 ===
* 13:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9cb9b3dfde}} (install toolforge_i18n from PyPI)

=== 2024-06-07 ===
* 09:06 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|253d1b0f45}} (l10n updates: pa)

=== 2024-05-26 ===
* 13:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|48a5585566}} (support opting out of Wikifunctions mode)

=== 2024-05-20 ===
* 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4d952df88b}} (l10n updates: ms)

=== 2024-05-13 ===
* 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1c3d80a5e6}} (l10n updates: eu, zh-hans)

=== 2024-05-11 ===
* 12:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bfccf1614c}} (more Hebrew verb templates)

=== 2024-05-09 ===
* 15:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b88dd1ce1}} (improve toolforge_i18n and upgrade dependencies for newer Babel and Werkzeug)

=== 2024-05-06 ===
* 17:04 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c5618f5968}} (set bot flag in bulk mode)
* 15:43 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8fa2740a72}} (README update, pulled without webservice restart)

=== 2024-05-05 ===
* 11:47 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|400cc9cb84}} (update Hebrew pa'al verbs)
* 11:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|19c8210d68}} (Hebrew pa'al verbs)

=== 2024-05-04 ===
* 12:17 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|deb5b1c44e}} (extract toolforge_i18n library: [[phab:T363626|T363626]])

=== 2024-05-03 ===
* 17:08 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|89c98da81f}} (upgrade dependencies for Python 3.12 compat; also upgraded pip&lt;nowiki&gt;{&lt;/nowiki&gt;,-tools&lt;nowiki&gt;}&lt;/nowiki&gt; and wheel while I’m at it)

=== 2024-04-22 ===
* 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1be060cd5c}} (l10n updates: ja)

=== 2024-04-18 ===
* 19:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f1a2cd1995}} (use public WikiLambda API)

=== 2024-04-17 ===
* 19:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e5d2281cea}} (l10n updates: krc)
* 18:13 wmbot~lucaswerkmeister@tools-bastion-13: pulled {{Gerrit|fa6c094165}} (templates CC BY-SA 3.0 → 4.0; no webservice restart needed)

=== 2024-04-08 ===
* 17:58 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|559eb5bc47}} (make session permanent after login)

=== 2024-04-06 ===
* 13:35 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|1569542ce6}} (l10n updates: el, fa, zh-hant)

=== 2024-03-24 ===
* 12:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|b630198d56}} (l10n updates: fi, ms-arab)

=== 2024-03-15 ===
* 19:41 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|272a303c09}} (Danish adverbs)
* 16:33 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8f4985e682}} (improve tests; should have no production impact but I pulled+restarted anyway ^^)

=== 2024-03-10 ===
* 18:40 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c62a9c1927}} (Maltese templates, including support for non-first forms to be the lemma: Maltese nouns have the third person singular as the lemma)
* 12:42 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bf88439696}} (l10n updates: fi, ko)

=== 2024-03-04 ===
* 18:12 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e7a659802c}} (l10n updates: ar, io, lb)

=== 2024-03-03 ===
* 00:26 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|4106259494}} (l10n updates: ht, hu)

=== 2024-02-28 ===
* 18:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|3030faaa3c}} (health-check-path, [[phab:T341919|T341919]])

=== 2024-02-23 ===
* 20:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|968078dbcd}} (l10n updates: hu, lt) [relog from 19:35 UTC, stashbot had problems]

=== 2024-02-17 ===
* 10:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|f88f2445fc}} (Esperanto adjective+verb Wikifunctions)

=== 2024-02-13 ===
* 18:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|85b6ec6534}} (l10n updates: ja, kaa)

=== 2024-02-07 ===
* 17:59 wmbot~lucaswerkmeister@tools-sgebastion-10: started webservice again (and patched the startup probe into it); took a while to come up but now it seems to be working
* 17:49 wmbot~lucaswerkmeister@tools-sgebastion-10: stopped webservice, restart wasn’t working so let’s try harder
* 17:45 wmbot~lucaswerkmeister@tools-sgebastion-10: restarted webservice, log was full of various errors

=== 2024-02-06 ===
* 20:39 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|344fd43224}} (update Breton noun Wikifunctions)

=== 2024-01-31 ===
* 19:13 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|604b43e316}} (l10n updates: it)

=== 2024-01-26 ===
* 19:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|249d9da0b7}} (l10n updates: id, kaa, ru, th)
* 00:22 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|886d99636e}} (more Esperanto noun Wikifunctions)

=== 2024-01-22 ===
* 18:34 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|0b062cafa9}} (Norwegian language name templates)

=== 2024-01-13 ===
* 15:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|d24dc99256}} (l10n updates: ar)

=== 2024-01-07 ===
* 13:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a97ab796ea}} (wikifunctions: first form from lemma, if missing)

=== 2024-01-06 ===
* 16:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea6b02ac57}} (Wikifunctions returning lists, Z11991→Z12689)

=== 2024-01-04 ===
* 12:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|82f5578b9a}} (l10n updates: ca, de, pl)

=== 2023-12-30 ===
* 15:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5baa3871d0}} (l10n updates: lb, zh-hans)

=== 2023-12-28 ===
* 10:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45b698823a}} (update Italian adjectives)
* 10:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e68d80748}} (i18n updates: uk)

=== 2023-12-17 ===
* 18:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7611d4e980}} (l10n updates: ia, krc, sv)

=== 2023-12-11 ===
* 18:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|424615e192}} (l10n updates: de, krc, lb, nl, pnb)

=== 2023-12-09 ===
* 16:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fdc7c853c4}} (update Breton noun Wikifunctions)

=== 2023-12-05 ===
* 19:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95ee032c68}} (l10n updates: ca, hno, io, it, pnb, sl, tr; i18n test improvements and fixes)

=== 2023-12-01 ===
* 19:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ba19a1cd5f}} (l10n updates: ja, sk, zh-hans)
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7acef657d0}} (update Croation noun Wikifunctions)

=== 2023-11-29 ===
* 17:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|54a614fd41}} (fix some spacing)

=== 2023-11-25 ===
* 12:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|171fc2ea54}} (l10n updates: br)

=== 2023-11-19 ===
* 16:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0416376e58}} (German masculine noun Wikifunctions)
* 15:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|11e7d12745}} (one more set of German neuter noun Wikifunctions)
* 13:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|442f510a5b}} (German neuter noun Wikifunctions)

=== 2023-11-18 ===
* 17:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8c123e032e}} (l10n updates: br, he, ko)

=== 2023-11-12 ===
* 17:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cc2cf0ceaf}} (l10n updates: bn, fa, fr, gl, it, lb, mk, nb, vi, zh-hans, zh-hant; yue removed, existing settings are automatically replaced with zh-hant)

=== 2023-11-04 ===
* 18:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|203bc87b5b}} (more German feminine noun Wikifunctions – m/n will follow later)
* 12:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfa1ad40e0}} (first German Wikifunctions: feminine noun -(e)n plural)
* 10:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|365c7e2814}} (cache Wikifunctions results)

=== 2023-11-01 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|240a228f49}} (tests for Wikifunctions, pulled without webservice restart)
* 18:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|92a91137e6}} (Wikifunctions for Breton nouns)

=== 2023-10-30 ===
* 19:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bea713bc0c}} (l10n updates: br)
* 00:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f33e56597c}} (update French Wikifunctions button label)

=== 2023-10-29 ===
* 17:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cca1b1af23}} (Wikifunctions support in edit mode)
* 16:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b5af35ab2b}} (fix Croatian feminine noun instrumental plural)
* 16:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|052ba84de7}} (fix crash for users without Wikifunctions account)
* 15:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c3dc0dd6d}} (experimental Wikifunctions for Esperanto nouns, nominative plural only)
* 14:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ab3c10890}} (fix Wikifunctions buttons lang= and dir=)
* 14:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5657d03fbb}} (experimental Wikifunctions for French nouns)
* 14:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a64b857485}} (experimental Wikifunctions for Croatian nouns)
* 14:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|40b0df49ee}} (experimental Wikifunctions support – happy birthday Wikidata 🎉)

=== 2023-10-28 ===
* 22:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c1f7a335e8}} (fix input patterns)

=== 2023-10-25 ===
* 17:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cdb1d34e11}} (Werkzeug 3.0.1)

=== 2023-10-20 ===
* 17:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|df7cf04757}} (i18n updates: io, ms-arab)

=== 2023-10-10 ===
* 19:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ad16425ee2}} (l10n updates: nl, uk, zh-hans)

=== 2023-10-06 ===
* 17:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72e12c5a2c}} (l10n updates: zh-hans) + remove hardcoded support for Karai-karai now that MediaWiki has it

=== 2023-10-01 ===
* 17:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|216afb45fa}} (update dependencies, Flask+Werkzeug 3)

=== 2023-09-24 ===
* 13:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5ae3295bb}} (Babel language code of Aragonese, to silence log warnings)
* 13:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45aa8fe43b}} (Danish proper nouns)

=== 2023-09-22 ===
* 16:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72c20b3b3e}} (l10n updates: cs, kai [new, with temporary hacks], tr, zh ⇒ zh-hans)

=== 2023-09-04 ===
* 16:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|85d978855f}} (Italian adverbs)

=== 2023-08-28 ===
* 18:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|48e3991eb6}} (fix typo in armenian-noun-singulare-tantum)

=== 2023-08-27 ===
* 14:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea49f8c2c7}} (update dependencies)
* 13:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|05522cee84}} (update Italian)

=== 2023-08-24 ===
* 17:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c19c9624ba}} (l10n updates: ca, fa, io)

=== 2023-08-12 ===
* 11:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e0cf031e70}} (l10n updates: it)

=== 2023-08-08 ===
* 18:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|56acd0944a}} (l10n updates: tr)

=== 2023-07-27 ===
* 12:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5d374c3787}} (l10n updates: ban, de, gl)

=== 2023-07-19 ===
* 12:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4fa53fae89}} (l10n updates: pt-br)

=== 2023-07-18 ===
* 08:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|474e48d752}} (update Breton grammatical feature)

=== 2023-07-15 ===
* 12:03 wm-bot: &lt;lucaswerkmeister&gt; pip-sync (i.e., actually install dependencies in the new venv, which I completely forgot to do earlier)
* 11:31 wm-bot: &lt;lucaswerkmeister&gt; kubectl patch deployment lexeme-forms --patch-file patch-add-startup-probe.yml
* 11:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|02f72f81a2}} (Python 3.11)

=== 2023-07-13 ===
* 13:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d7fea069ba}} (l10n updates: pl)

=== 2023-07-10 ===
* 17:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|42679bb5dc}} (l10n updates: yue)

=== 2023-07-09 ===
* 14:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|78711ad373}} (l10n updates: ms)

=== 2023-07-02 ===
* 13:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e2653cf19}} (revert recent punjabi-noun-masculine-guru change)
* 12:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c9e84dfb8d}} (add separators to Dutch nouns)

=== 2023-06-30 ===
* 18:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1ed453b5d5}} (l10n updates: sh → sh-latn, tt → tt-cyrl, [[phab:T336606|T336606]])

=== 2023-06-27 ===
* 20:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fe5983c571}} (l10n updates: ba)

=== 2023-06-25 ===
* 14:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3ad131b7bf}} (Aragonese common nouns)

=== 2023-06-24 ===
* 09:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|213bfabfb4}} (underline links on hover again)

=== 2023-06-22 ===
* 20:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|63c042d9b3}} (l10n updates: it)

=== 2023-06-20 ===
* 18:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7081d2769e}} (support language fallback and ?uselang)
* 17:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e76345eb5}} (l10n updates: ba, id, nb, xmf)

=== 2023-06-18 ===
* 11:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bebc116e22}} (Bootstrap 5)
* 11:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d53e455ef7}} (update Malayalam nouns and add adjective template)

=== 2023-06-16 ===
* 17:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|248590aeb0}} (l10n updates: ba, id, pl)

=== 2023-06-13 ===
* 17:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e9112d022e}} (l10n updates: es)

=== 2023-06-11 ===
* 11:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fb8c4a30ff}} (update punjabi-noun-masculine-guru)

=== 2023-06-09 ===
* 16:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e059c8bbd6}} (l10n updates: fi); also, last time I forgot to git rebase, so this actually includes {{Gerrit|2035050d28}} (l10n updates: sv) as well

=== 2023-06-07 ===
* 07:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2035050d28}} (l10n updates: sv)

=== 2023-06-04 ===
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|08962e4902}} (update past transgressive item ID after merge; only affects czech-verb-perfective)

=== 2023-05-31 ===
* 20:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1ec8c72304}} (Russian adjectivse: remove compound lexical categories)

=== 2023-05-29 ===
* 15:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a5e90a0e02}} (update dependencies)

=== 2023-05-27 ===
* 19:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|07deb7a083}} (Punjabi additive double causative verbs)
* 17:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|889b4ce276}} (Punjabi additive causative verbs)
* 15:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|467d5b9f34}} (Punjabi transitive verbs)
* 15:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6c76e2d3b5}} (fix two Punjabi placeholders)

=== 2023-05-25 ===
* 20:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a50e668166}} (l10n updates: ca, es, fa, fi, ru, tr, ur)

=== 2023-05-19 ===
* 17:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b59c2f0aad}} (l10n updates: es, hi, zh-hant)
* 16:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b80c8ff9db}} (fix “logged in” indicator in several languages)

=== 2023-05-18 ===
* 08:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7f76b0f203}} (l10n updates: br, de, fr, he, hi, hno, ia, mk, pa, pnb, ru, sa, sl, ur)

=== 2023-05-13 ===
* 17:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7d3ab49b06}} (l10n updates: ar, bn, de, eo, fa, fi, fr, he, hy, ia, it, ja, ko, mk, ms, nb, pnb, ru, skr-arab, sl, zh-hant)
* 12:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dfcf34ed51}} (make “logged in as” translatable)
* 11:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|65cb94f3c7}} (punjabi-verb-basic-intransitive templates)

=== 2023-05-12 ===
* 20:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|15b7403971}} (fix stray character)

=== 2023-05-08 ===
* 21:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db3dd67b8a}} (make more translations available and tweak Babel language codes)
* 20:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7bf757be9}} (fix message keys broken by previous deployment)
* 20:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5f01c59794}} (refactor message keys from _ to -, should make no difference)
* 19:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b51f930220}} (user interface language setting)

=== 2023-05-05 ===
* 12:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72a006c6ea}} (l10n updates: mrh, ta)

=== 2023-05-02 ===
* 00:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5f83647d21}} (test-only change, pulled without webservice restart)

=== 2023-05-01 ===
* 23:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|88da33ddc5}} (GitHub actions only change, pulled without webservice restart)
* 17:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1380884cce}} (upgrade dependescies, GHSA-m2qf-hxjv-5gpq)
* 15:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|75230357a4}} (l10n updates: lt)
* 15:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1554678038}} (improve matching.py for upcoming templates, should make no difference at the moment)
* 14:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|692e255a50}} (refactor matching.py, should make no difference)

=== 2023-04-30 ===
* 15:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db66a9373c}} (refactor statement groups; should make no difference)

=== 2023-04-25 ===
* 21:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9059e45cda}} (update dependencies, Werkzeug 2.3.0 / Flask 2.3.1)
* 18:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c6dc908e1e}} (refactoring for somevalue support, should make no difference yet)

=== 2023-04-24 ===
* 19:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d5b5c8994f}} (preparation &amp; refactoring, no visible changes)

=== 2023-04-23 ===
* 18:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0f96d60736}} (Punjabi adverbs)
* 18:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|75af96b851}} (Punjabi adjectives)
* 15:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|934f5cffdb}} (Yoruba adjectives)

=== 2023-04-22 ===
* 16:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a074fd9c64}} (trim spaces)
* 15:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fdb0552957}} (remove spaces)
* 15:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b6a1268b21}} (Punjabi nouns)

=== 2023-04-15 ===
* 15:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|604df5c72e}} (two more variables)
* 15:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1b999f4661}} (use variables for entity IDs; should make no difference at runtime)
* 14:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|24fb20fd19}} (sort sets for JSON output)

=== 2023-04-12 ===
* 20:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5b07592a7e}} (two style improvements)

=== 2023-04-10 ===
* 17:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|282a7b6b18}} (l10n updates: anp; currently skipped because unsupported by Babel)

=== 2023-04-08 ===
* 11:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|994cbd48b0}} (fix typo in a Hindustani template)

=== 2023-04-01 ===
* 18:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|08ac04d468}} (fix Hindko template order)

=== 2023-03-22 ===
* 20:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b40cefa378}} (change Hindko templates to hno)

=== 2023-03-19 ===
* 21:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cf1e031a43}} (l10n updates: fi, tt)

=== 2023-03-13 ===
* 21:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d7ba3ddc23}} (l10n updates: hi, pa, tt, ur)

=== 2023-03-08 ===
* 22:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8da3525baf}} (fix lowercase item ID in portuguese-noun-biform)
* 22:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|de17c6bdf6}} (fix hindustani-verb-additive-causative-double-ur label)
* 22:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a99078e1c5}} (hindustani-verb-additive-causative-double templates)

=== 2023-03-06 ===
* 21:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8828e3269e}} (l10n updates: tt)
* 21:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2cbf107d6e}} (hindustani-verb-additive-causative templates)
* 20:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0f7a634e72}} (fix Hindustani verb placeholders)

=== 2023-03-05 ===
* 21:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c325634bc3}} (hindustani-verb-additive-transitive templates)
* 19:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|85cbe15d08}} (hindustani-verb-basic-transitive templates)
* 13:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|00f87cf139}} (hindustani-verb-basic-intransitive templates)

=== 2023-03-03 ===
* 20:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c310fd9d88}} (update Hindustani labels, and l10n update: tt)

=== 2023-02-27 ===
* 19:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|50aa1e2dc5}} (l10n updates: hi, hno, pa, pnb, ur)

=== 2023-02-26 ===
* 21:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1d6c0caecd}} (Hindustani non-verb templates – verbs still TBD, need more time)
* 15:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2feff85812}} (use hno translations)

=== 2023-02-22 ===
* 20:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9e667986b4}} (l10n updates: hi, ur)

=== 2023-02-14 ===
* 19:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9debac9385}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes; venv rebuilt from scratch to avoid NFS issues)
* 19:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfd63ebac1}} (l10n updates: hno); also, turns out I didn’t git rebase in the last deployment, so this *actually* deploys the Danish nouns update and pl l10n update

=== 2023-02-09 ===
* 20:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2912ebfa68}} (update Danish nouns, and l10n updates: pl)

=== 2023-01-31 ===
* 19:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfaf13f447}} (update github actions; pulled without webservice restart)
* 19:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f9bf85df5f}} (l10n updates: cy)

=== 2023-01-29 ===
* 12:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3ca9650fe1}} (Danish adjectives)

=== 2023-01-09 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4857d874ce}} (l10n updates: pa)

=== 2023-01-03 ===
* 15:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c27eaec33}} (l10n updates: pl)

=== 2022-12-30 ===
* 12:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95b9026d22}} (l10n updates: pa, zh)

=== 2022-12-28 ===
* 15:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c47032838}} (fix bulk result display when given lexeme ID)

=== 2022-12-26 ===
* 11:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b51ddc8c08}} (update Armenian noun templates)
* 11:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bdaa43aef3}} (preserve target_hash in more places)

=== 2022-12-16 ===
* 21:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4802902384}} (l10n updates: yue)

=== 2022-12-08 ===
* 19:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3f6b15c1f0}} (l10n updates: fa, gl, pl, sl)

=== 2022-12-06 ===
* 13:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|97001e468b}} (fix missing statements)
* 13:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45a026916c}} (fix Hindko feminine noun template)

=== 2022-12-05 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4d781fb933}} (Hindko noun templates)
* 20:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2cb7ac792f}} (l10n updates: pnb)

=== 2022-12-04 ===
* 17:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|82a2272a2f}} (three new Norwegian Nynorsk noun templates)

=== 2022-11-29 ===
* 21:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b0ebae4629}} (l10n updates: el)

=== 2022-11-27 ===
* 19:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bb7cf271ae}} (l10n updates: fa)

=== 2022-11-19 ===
* 15:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|10af55574b}} (more Bokmål and Nynorsk templates)

=== 2022-11-15 ===
* 20:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5897fd06ee}} (Danish nouns fix)
* 20:23 wm-bot: &lt;lucaswerkmeister&gt; ionice -c3 zstd --rm uwsgi.log.1668543276 # 8.85%, {{Gerrit|520591680}} =&gt; {{Gerrit|46091850}} bytes)
* 20:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2b53b1199c}} (rotate uwsgi.log after 100 MiB)
* 19:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0429d7d80b}} (update Danish nouns+verbs)

=== 2022-11-10 ===
* 13:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5160edb9ca}} (l10n updates: pnb)
* 13:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|127e065522}} (NFC-normalize lemma for search)

=== 2022-11-07 ===
* 21:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0c7095c96d}} (Polish adjectives, positive only)
* 20:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c653fb07e2}} (l10n updates: es, hy, pnb)

=== 2022-11-05 ===
* 14:02 wm-bot: &lt;lucaswerkmeister&gt; git gc (.git 19M → 1.1M)
* 13:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8feb3f86d4}} (extra GitHub actions job, pulled without webservice restart)
* 12:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d38d5ba55c}} (uninstall dev dependencies in production; reduces venv size from ca. 142 MB to ca. 75 MB, or about by half)
* 12:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b7f4d4ba31}} (added test; pulled without webservice restart)
* 11:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ccecd3bb87}} (l10n updates: krc, zh)

=== 2022-10-27 ===
* 12:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03b6dd3b71}} (l10n updates: pnb)

=== 2022-10-26 ===
* 20:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2feba604c7}} (update dependencies, use PEP 655 NotRequired)
* 19:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|55f9b203e5}} (l10n updates: sl)

=== 2022-10-23 ===
* 16:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3844a7df05}} (French verbs)

=== 2022-10-17 ===
* 19:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b098904d43}} (l10n updates: ja, pnb)

=== 2022-10-14 ===
* 18:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a829f83124}} (l10n updates: ca, hi, sh, sl)

=== 2022-10-05 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|953b553968}} (translate Hebrew adjective template label)
* 18:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|93ebb772c5}} (more Spanish templates)

=== 2022-10-01 ===
* 19:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b770688eb1}} (Hebrew adjectives)
* 18:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8137259ca6}} (Flask 2.2)

=== 2022-09-23 ===
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c66922341d}} (l10n updates: ar, ku, sl)

=== 2022-09-18 ===
* 16:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8d996c1fa4}} (l10n updates: ar)

=== 2022-09-10 ===
* 18:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|609066f02b}} (README fix, pulled without webservice restart)
* 16:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|52570991cd}} (diffusion → gitlab)

=== 2022-08-29 ===
* 20:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fa8f5d87a4}} (l10n updates)

=== 2022-08-25 ===
* 14:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|019b4ecc79}} (optimize messages with unused GENDER magic word)
* 14:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dd6cb7f08b}} (l10n updates)

=== 2022-08-03 ===
* 19:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a11b6a55f6}} (l10n updates)

=== 2022-07-21 ===
* 23:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|38141487d1}} (l10n updates)

=== 2022-07-17 ===
* 17:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|238f943e8a}} (add more typing; hopefully no functional changes)

=== 2022-07-13 ===
* 20:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d5cb20368d}} (l10n updates)

=== 2022-07-02 ===
* 19:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d3e2185bbc}} (l10n updates)

=== 2022-06-29 ===
* 19:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6ac757a997}} (Igbo verbs + pronouns)

=== 2022-06-16 ===
* 21:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|466976ba49}} (l10n updates)

=== 2022-06-14 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b0143851e0}} (l10n updates)

=== 2022-05-26 ===
* 20:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|24d9b273c5}} (l10n updates)

=== 2022-05-17 ===
* 19:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8cdef0cf20}} (l10n updates)

=== 2022-05-03 ===
* 20:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d8429a8740}} (l10n updates)

=== 2022-04-29 ===
* 19:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fd45333563}} (l10n updates, extra unit test)

=== 2022-04-28 ===
* 23:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|860abb205b}} (Bokmål passive verbs)

=== 2022-04-27 ===
* 20:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c92b363387}} (Mandarin templates)

=== 2022-04-25 ===
* 19:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7b5d0d7298}} (l10n updates)

=== 2022-04-22 ===
* 11:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d769b4ed8b}} (l10n updates)

=== 2022-04-20 ===
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|89a5273967}} (l10n updates)

=== 2022-04-15 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; pulled {{Gerrit|24d5774c5f}} (test-only change, so no restart)
* 18:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|54a5376631}} (update German verbs)
* 16:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9a2cefe8e6}} (updated Portuguese templates)

=== 2022-04-04 ===
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|197baf2940}} (l10n updates)

=== 2022-03-30 ===
* 18:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c6001bf897}} (l10n updates; use pip-tools, includes some package updates such as Flask 2.0.2→2.1.0; clean up service.template)

=== 2022-03-19 ===
* 12:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f573b558d4}} (l10n updates)

=== 2022-03-11 ===
* 00:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d7787d7536}} (l10n updates)

=== 2022-03-05 ===
* 18:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72f2adc394}} (l10n updates)

=== 2022-02-28 ===
* 12:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|04ba7580ab}} (l10n updates)

=== 2022-02-25 ===
* 00:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1506d1a9e9}} (l10n updates)

=== 2022-02-22 ===
* 00:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1fc2f98450}} (l10n updates)

=== 2022-02-15 ===
* 13:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|56e69bad1a}} (l10n updates)

=== 2022-02-11 ===
* 23:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b4624e0bbc}} (l10n updates)

=== 2022-02-07 ===
* 13:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b3c5446831}} (l10n updates)

=== 2022-01-30 ===
* 12:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c1d6a79ed2}} (update Odia nongendered adjectives)

=== 2022-01-22 ===
* 17:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b1cc42ef84}} (Odia nouns)
* 16:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b62723fb6f}} (update Odia adverbs)

=== 2022-01-16 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|504c5481e9}} (update Spanish verbs)
* 18:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|68234bd17d}} (Odia adjectives and adverbs)

=== 2022-01-10 ===
* 18:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d1da801731}} (l10n updates)

=== 2022-01-06 ===
* 18:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57dc392b8f}} (l10n updates)

=== 2022-01-03 ===
* 18:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aacaae3cd6}} (revert update of indefinite item ID after merge, I flipped the items)
* 15:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2eb6822ed2}} (l10n updates)
* 15:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7312514fc8}} (update indefinite item ID after merge)

=== 2022-01-01 ===
* 23:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d6110ed631}} (l10n updates)

=== 2021-12-17 ===
* 21:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|20c4392de6}} (l10n updates)

=== 2021-12-02 ===
* 23:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2a2cb9b211}} (l10n updates)

=== 2021-11-25 ===
* 21:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|baef3a16f6}} (l10n updates)

=== 2021-11-18 ===
* 13:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e001c252c5}} (l10n updates, including initial Yoruba translations)

=== 2021-11-14 ===
* 14:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c113d4dd77}} (Yoruba nouns)

=== 2021-11-08 ===
* 22:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|85719cf3ae}} (update Portuguese idioms)
* 22:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e58c43ab3e}} (Portuguese idioms quickfix)

=== 2021-11-07 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|91216ed64b}} (Portuguese idioms)

=== 2021-11-06 ===
* 12:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7ef5eb34a3}} (fix Manbhumi bulk mode link)

=== 2021-11-04 ===
* 12:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d649d7a24a}} (l10n updates)

=== 2021-10-25 ===
* 19:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0f5b5de66a}} (bump startupProbe failureThreshold 3→10)
* 19:34 wm-bot: &lt;lucaswerkmeister&gt; deployment was successful after all 🤷
* 19:31 wm-bot: &lt;lucaswerkmeister&gt; belay that, the new pod hasn’t actually started properly. investigating
* 19:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|754342b9a3}} (language name for bn-x-Q6747180)

=== 2021-10-18 ===
* 12:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eae6c8d594}} (l10n updates)

=== 2021-10-16 ===
* 14:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1903c3d0eb}} (don’t show duplicate warning errors)
* 12:09 wm-bot: &lt;lucaswerkmeister&gt; pulled {{Gerrit|8700382f98}} (rename confusingly named deplyoment patch file) without webservice restart
* 12:04 wm-bot: &lt;lucaswerkmeister&gt; (correction on that last message, it’s a startup probe now, not a readiness probe)
* 12:03 wm-bot: &lt;lucaswerkmeister&gt; patched readiness probe into deployment again
* 12:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19fb8c90ee}} (findDuplicates fix) with full stop/start to pick up label changes

=== 2021-10-13 ===
* 23:31 wm-bot: &lt;lucaswerkmeister&gt; fully restarted webservice (stop/start) to avoid label issues
* 17:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5c87ff53c}} (remove type ignore comments) and updated dependencies, including Flask 2.0.2

=== 2021-10-11 ===
* 12:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fb32d04132}} (l10n updates)

=== 2021-10-10 ===
* 11:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bf2834c472}} (improve error handling)

=== 2021-10-04 ===
* 19:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1697521bf5}} (l10n updates)

=== 2021-09-25 ===
* 14:45 wm-bot: &lt;lucaswerkmeister&gt; removed old venv-3.7
* 13:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6f9e530018}} (mobile-friendly navbar)
* 13:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea93caf2ee}} (l10n updates)

=== 2021-09-19 ===
* 13:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c1b6e0810}} (readinessProbe → startupProbe to avoid bloating access log); deployed by adding readinessProbe: null to the patch file and patching the deployment with that

=== 2021-09-14 ===
* 20:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c36ae4154a}} (l10n updates)
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|902156ddb8}} (Croatian item ID fix)

=== 2021-09-12 ===
* 21:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4da7f64c4b}} (updates without downtime)
* 20:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f21554ab71}} (refactoring, noop)
* 15:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a4b05045d6}} (Croatian nouns)

=== 2021-09-08 ===
* 20:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2aa32a0f7f}} (l10n updates)

=== 2021-09-03 ===
* 15:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3698f0b79c}} (add passive forms to Norwegian Bokmal verbs)
* 15:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8051248b60}} (l10n updates)

=== 2021-08-30 ===
* 18:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dfc0838301}} (l10n updates)

=== 2021-08-25 ===
* 20:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|237a5414d5}} (l10n updates)

=== 2021-08-19 ===
* 20:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bcc4c3aa63}} (l10n updates)

=== 2021-08-17 ===
* 21:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ca42b7cdb}} (more types)
* 18:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2382c30c01}} (initial mypy setup)
* 17:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c66572938e}} (python3.9)

=== 2021-08-16 ===
* 12:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|92e5e0d70c}} (l10n updates)

=== 2021-08-14 ===
* 12:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7a1980f4e2}} (l10n updates)

=== 2021-08-11 ===
* 19:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|37acc67c90}} (l10n updates)

=== 2021-08-02 ===
* 19:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|de5ab0e740}} (l10n updates)

=== 2021-07-19 ===
* 18:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0c9f1015c0}} (work around Firefox bug)

=== 2021-07-18 ===
* 18:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fa64f7e021}} (refuse to load non-user-readable config file, guard against recurrence of [[phab:T286414|T286414]])
* 13:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61b1d0fd93}} (Igbo adjectives and fix nouns)

=== 2021-07-17 ===
* 11:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0d1f3d924e}} (load config file differently)

=== 2021-07-16 ===
* 19:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|37766a8002}} (l10n updates)

=== 2021-07-11 ===
* 20:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5dbc39eb5e}} (l10n update)
* 17:03 wm-bot: &lt;lucaswerkmeister&gt; restarted webservice to pick up 1.3 version of OAuth consumer ([[phab:T286414|T286414]])
* 13:36 wm-bot: &lt;lucaswerkmeister&gt; chmod go-rwx www/python/src/config.yaml # [[phab:T286414|T286414]]

=== 2021-07-01 ===
* 23:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ac8779515d}} (l10n updates)
* 23:37 wm-bot: &lt;lucaswerkmeister&gt; unlink ~/services.template # new version of webservice doesn’t like the symlink :(

=== 2021-06-28 ===
* 17:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|64c5584c9d}} (remove workaround for [[phab:T241422|T241422]])
* 17:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5565da07e5}} (l10n updates, especially Igbo translations)

=== 2021-06-22 ===
* 19:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c88b1962fa}} (Igbo nouns)

=== 2021-06-21 ===
* 20:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19098277f4}} (l10n updates)

=== 2021-06-20 ===
* 12:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|afc6f6f242}} (update German verbs)

=== 2021-06-19 ===
* 19:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c5b12d5dc1}} (Malayalam proper nouns)
* 19:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|05cd31e9bd}} (update Malayalam noun)

=== 2021-06-15 ===
* 20:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0b6fed0054}} (even more optional grammatical features)
* 19:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d8eadd1cae}} (more optional grammatical features)
* 18:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61a5e0fc18}} (optional grammatical features)

=== 2021-06-14 ===
* 23:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|626b73a005}} (l10n updates)
* 23:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|70efbdc1a7}} (update volitive item ID)

=== 2021-06-10 ===
* 20:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1f94df1209}} (l10n updates)

=== 2021-06-07 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|547231388b}} (add create link for duplicates in bulk mode)
* 20:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|daf88503e0}} (l10n updates)

=== 2021-06-06 ===
* 14:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2040a7497e}} (target_hash URL parameter)

=== 2021-06-05 ===
* 20:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fcf67b1016}} (improve title)

=== 2021-06-04 ===
* 23:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|16c0cd2606}} (improve batch mode results page)

=== 2021-05-31 ===
* 20:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|43a29c4369}} (replace deprecated function)
* 20:00 wm-bot: &lt;lucaswerkmeister&gt; pip upgrade (Flask 2.0.1 and other updates)
* 19:59 wm-bot: &lt;lucaswerkmeister&gt; briefly stopping tool to upgrade venv
* 18:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|148dafa60b}} (l10n updates)

=== 2021-05-30 ===
* 14:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c047f6aca}} (l10n updates)

=== 2021-05-24 ===
* 18:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6ffd1a2c1b}} (update Esperanto verb)
* 16:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7d43094e56}} (l10n updates)
* 11:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e0099e68d5}} (Swedish adjective)

=== 2021-05-22 ===
* 09:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|31e85bafcf}} (l10n updates)
* 09:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|44812d4446}} (add Portuguese modal adverb)

=== 2021-05-15 ===
* 14:01 wm-bot: &lt;lucaswerkmeister&gt; tool should be back up (uwsgi.log went from 181M to 77M after moving pre-2021 data to separate files)
* 13:56 wm-bot: &lt;lucaswerkmeister&gt; briefly stopping tool (few minutes) to cycle the uwsgi.log

=== 2021-05-13 ===
* 23:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e2ceb0513}} (l10n updates)
* 14:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|67e7cf3dfb}} (rename Swedish adjective template)
* 13:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95f40ac9d5}} (Norwegian Bokmål masculine/neuter nouns)

=== 2021-05-10 ===
* 16:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|248527544d}} (l10n updates)

=== 2021-05-09 ===
* 13:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5951b46450}} (fix lang= and dir= on index)

=== 2021-05-03 ===
* 19:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b159dd1060}} (l10n updates)

=== 2021-05-02 ===
* 11:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4c9a5f0ebf}} (duplicate check JS fixes)

=== 2021-05-01 ===
* 14:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61744950f0}} (l10n updates)

=== 2021-04-26 ===
* 19:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|abf6719d31}} (Python 3.7 fix)
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d15d0c5f2d}} (rename Dutch templates)
* 18:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|868ee95cf2}} (l10n updates)

=== 2021-04-22 ===
* 19:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8ab4ceb62a}} (l10n updates)

=== 2021-04-19 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2f8f589a62}} (Swedish proper nouns)
* 20:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4effbc2a36}} (l10n updates)

=== 2021-04-17 ===
* 10:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1d10ab467e}} (fix bulk mode)

=== 2021-04-15 ===
* 19:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|051e3789a2}} (l10n updates)

=== 2021-04-14 ===
* 20:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b17ed175fe}} (move login hint up)
* 20:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0006696173}} (remove automatic login redirect)
* 12:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|30c561955f}} (login link in navbar)

=== 2021-04-12 ===
* 18:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e4682a00bd}} (Breton noun fixes)
* 18:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a3a81d0c4b}} (l10n updates)

=== 2021-04-09 ===
* 18:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|18bb25abd0}} (l10n updates)

=== 2021-04-05 ===
* 13:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f5439f66a2}} (l10n updates)

=== 2021-04-04 ===
* 13:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9507991400}} (Malayalam verb fix)

=== 2021-04-03 ===
* 19:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e2bc5b577}} (language code refactorings; should not result in any observable changes)
* 18:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8416f8d861}} (more Breton nouns + adverbs)
* 16:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|21201880f5}} (MarkupSafe-aware formatters; should not result in any observable changes)
* 15:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|615bba5934}} (better bulk mode errors)

=== 2021-04-02 ===
* 19:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|be73b49e29}} (better language code handling)

=== 2021-04-01 ===
* 18:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f2b128273d}} (l10n updates)

=== 2021-03-30 ===
* 21:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7ff57d504e}} (l10n updates)

=== 2021-03-28 ===
* 19:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|43d0c29996}} (update Portuguese nouns)
* 14:16 wm-bot: &lt;lucaswerkmeister&gt; &lt;em&gt;actually&lt;/em&gt; deployed {{Gerrit|2ece3adc91}} (this time I did the &lt;code&gt;git rebase&lt;/code&gt; but forgot the &lt;code&gt;webservice restart&lt;/code&gt;, how’s that for a change)
* 13:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2ece3adc91}} (Portuguese updates)

=== 2021-03-27 ===
* 14:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1f2a6f2e17}} (replace OrderedDict with dict)
* 13:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4619f8cd03}} (remove duplicate template)
* 13:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9ad3addd6a}} (Malayalam verbs, and vocative case for nouns)

=== 2021-03-26 ===
* 21:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5b44b44f52}} (Malayalam verbs)
* 21:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|78a5c9a10a}} (indicate optional forms)

=== 2021-03-25 ===
* 19:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|77328e559d}} (optional forms)

=== 2021-03-24 ===
* 22:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ffa45a58b1}} (minifix)
* 19:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea6928faaa}} (clarify Norwegian Bokmål adjectives)
* 19:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|99257d861c}} (Portuguese adjectives)

=== 2021-03-23 ===
* 21:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|253aed283c}} (Latvian nouns)
* 19:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c0b2c473ff}} (add language code as ID on index page, suggested by jhsoby)

=== 2021-03-22 ===
* 21:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2e4e3dca5a}} (improved Malayalam nouns [not verbs as it says in the commit message, oops] + i18n updates)

=== 2021-03-16 ===
* 19:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|547b42f25f}} (Portuguese nouns, i18n updates)

=== 2021-03-13 ===
* 16:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f389caf9b2}} (gender i18n improvements, should be a no-op)

=== 2021-03-12 ===
* 20:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9500beeed4}} (three new translations) – should be a no-op but I didn’t want to leave it lying around without a webservice restart either
* 19:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aa07bef3bd}} (i18n update) – also, previous SAL message mentioned {{Gerrit|712d262475}} but that’s still in &lt;code&gt;git log @..@&lt;nowiki&gt;{&lt;/nowiki&gt;u&lt;nowiki&gt;}&lt;/nowiki&gt;&lt;/code&gt;, so I think I forgot to rebase last time

=== 2021-03-10 ===
* 20:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|712d262475}} (restore logging for generic API errors)
* 19:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|94dfecbc2a}} (generic API error handler)

=== 2021-03-08 ===
* 14:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b7b55e1b33}} (more i18n improvements)
* 11:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea7cd3ac71}} (i18n from translatewiki.net – [[phab:T272243|T272243]])

=== 2021-03-05 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|109f22a415}} (Czech verbs update)

=== 2021-03-04 ===
* 21:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1435d31446}} (update Swedish translations)
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|15a24d63eb}} (minor Czech verbs improvement)

=== 2021-02-28 ===
* 17:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|369031b945}} (minifix)
* 17:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0455dc20f4}} (better OAuth error handling)

=== 2021-02-19 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f66f631598}} (auth improvements)

=== 2021-02-18 ===
* 20:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a0ba7b84ab}} (quickfix)
* 20:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|23ccbcf6f6}} (work around [[phab:T272319|T272319]])

=== 2021-02-16 ===
* 20:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8d96af0ec2}} (add skip link)
* 19:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e716e6d6d}} (Bootstrap update)

=== 2021-02-13 ===
* 22:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|02a2edf583}} (edit summary fixes)
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7257a065e}} (code style fixes)
* 16:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e70e759d7}} (minifix)
* 13:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fb17f5e4ef}} (edit mode fix for forms with multiple representations)

=== 2021-02-11 ===
* 22:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|81166d5c17}} (reduce [[phab:T230833|T230833]] workaround / &quot;und&quot; language codes)
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8e718af67e}} (JS fix)

=== 2021-02-10 ===
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0d8279ca7f}} (&lt;script&gt; loading improvements)
* 20:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1fe3d3589e}} (prevent double submit)

=== 2021-02-04 ===
* 20:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|32b6b23f72}} (German adverbs)

=== 2021-02-01 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f4e7ba98a7}} (stop referrer-URL comparison)
* 14:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d237952e44}} (fix current_url / CSRF detection)

=== 2021-01-30 ===
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a87ce138db}} (show bulk parse errors)

=== 2021-01-28 ===
* 20:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|868bccbbe7}} (fall back to en)
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cb0855af48}} (simplify current_url)

=== 2021-01-27 ===
* 22:39 wm-bot: &lt;lucaswerkmeister&gt; deployed fixed version of test code, oops
* 22:38 wm-bot: &lt;lucaswerkmeister&gt; deployed another version of test code
* 22:26 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted test code to print current_url debug output
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1bc8d4232e}} (remove long-dead code about fixing the session cookie)
* 20:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03255e1408}} (pop OAuth redirect target)

=== 2021-01-13 ===
* 20:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5725705d1}} (fix edit mode, drop form data stashing)

=== 2021-01-09 ===
* 21:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9a604413d3}} (German toponym)

=== 2021-01-07 ===
* 14:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|00d7fe313e}} (better edit links)

=== 2021-01-03 ===
* 11:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db1e890252}} (grab cursor for draggable links)

=== 2020-12-30 ===
* 12:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|191518cbf9}} (edit lemma when adding first form)

=== 2020-12-23 ===
* 15:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6d8bae537b}} (Esperanto verb)
* 14:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|69f610af18}} (Breton noun, without mutation, collective)

=== 2020-12-22 ===
* 11:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6e1185532d}} (Basque adjective)

=== 2020-12-14 ===
* 20:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9ba55b3ad3}} (fix current_url)

=== 2020-12-13 ===
* 00:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bb0cbfc6cb}} (language code in parentheses)

=== 2020-12-12 ===
* 18:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ec650ea2f}} (autonyms on index page)

=== 2020-12-02 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5291d5cda}} (more Esperanto translations)

=== 2020-11-29 ===
* 21:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|915eb4016f}} (clarify German templates)

=== 2020-11-24 ===
* 21:58 wm-bot: &lt;lucaswerkmeister&gt; undeployed debug code, I don’t remember what it was for anymore
* 21:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|59f2c38fed}} (the previously-uncommitted JS fix, now committed; some uncommitted debug code is still there)

=== 2020-11-21 ===
* 21:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1608cc4dd9}} (gender-dependent messages)

=== 2020-11-05 ===
* 19:51 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted JS fix, to be committed later if it works as intended

=== 2020-10-29 ===
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1a150904fd}} (update Italian translations)

=== 2020-10-26 ===
* 21:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e3c4c2e664}} (Esperanto adjective)

=== 2020-10-25 ===
* 21:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bd4c445f02}} (edit mode fix)
* 21:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|782dfdabee}} (fixes for edit mode and ordia links)

=== 2020-10-24 ===
* 13:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|792db2a9f9}} (edit mode language_code parameter)

=== 2020-10-19 ===
* 20:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7fd004ef9}} (drag’n’drop fix; submit_lexeme debug code still there)

=== 2020-10-17 ===
* 14:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19b5bc257a}} (more durable CSRF tokens; some uncommitted debug code to print submit_lexeme errors is still there)

=== 2020-10-08 ===
* 20:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fd8c692798}} (fix a crash; debug code still in place)

=== 2020-09-13 ===
* 08:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9f02b375f1}} (more conventient bulk mode transition; debug code still present)
* 08:17 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted extra logging for submit_lexeme errors in bulk mode

=== 2020-09-12 ===
* 12:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ce943856ed}} (fix Spanish feminine noun item ID)

=== 2020-09-08 ===
* 16:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9ac796e7aa}} (Manbhumi verbs)

=== 2020-09-06 ===
* 08:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|116e4123b0}} (fix Manbhumi duplicate search)

=== 2020-09-01 ===
* 15:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ef72c06ec8}} (Manbhumi adjectives and adverbs)

=== 2020-08-14 ===
* 19:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|13282d5404}} (Bengali verb updates)

=== 2020-08-12 ===
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e3291c8796}} (Bengali adverbs, other improvements)

=== 2020-08-04 ===
* 22:43 wm-bot: &lt;lucaswerkmeister&gt; &lt;em&gt;actually&lt;/em&gt; deployed {{Gerrit|39457a18ab}} (forgot to git rebase)
* 22:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|39457a18ab}} (Bengali adjectives and verbs)

=== 2020-07-08 ===
* 21:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b65c1018ff}} (translation update)

=== 2020-07-05 ===
* 22:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f29663c2b2}} (Norwegian Bokmål nouns)

=== 2020-07-04 ===
* 16:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cbf5ad6440}} (Norwegian Bokmål)

=== 2020-06-17 ===
* 23:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9b7349c602}} (update a Bengali template)

=== 2020-06-15 ===
* 20:54 wm-bot: &lt;lucaswerkmeister&gt; renamed default branch from master to main

=== 2020-06-14 ===
* 12:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8d5f428c3e}} (improved duplicate warning edit links)
* 10:15 wm-bot: &lt;lucaswerkmeister&gt; *actually* deployed {{Gerrit|2efe64f7e5}} (forgot to git rebase)
* 10:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2efe64f7e5}} (link edit mode in duplicate warning)

=== 2020-06-13 ===
* 21:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b42e79e6bb}} (more sections)
* 17:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cf1079fda1}} (more section improvements)
* 13:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c2e6d57a29}} (improved German sections)
* 11:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4cd36a71a1}} (sections in edit mode)
* 11:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e288f0106}} (sections)
* 08:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfa46d522b}} (Czech edit mode translations)

=== 2020-06-07 ===
* 20:53 wm-bot84: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9e4f3a1b65}} (two translation fixes)
* 13:35 wm-bot84: &lt;lucaswerkmeister&gt; deployed {{Gerrit|09cc2017ec}} (Bengali nouns)

=== 2020-05-24 ===
* 13:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c6d1c6e30}} (update Breton)

=== 2020-05-13 ===
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a2deb7908c}} (update past participle item ID after merge)

=== 2020-05-11 ===
* 19:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ddac27d2e2}} (translation update)

=== 2020-05-10 ===
* 22:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b797c90917}} (Breton typofix)
* 15:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eac96e8493}} (Breton adjectives and other improvements)
* 11:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fc78831f8e}} (Breton nouns)

=== 2020-05-09 ===
* 19:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b4780fa832}} (drag’n’drop unmatched forms in edit mode)

=== 2020-04-25 ===
* 20:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0dadbb4d4e}} (toolforge.org)

=== 2020-04-21 ===
* 21:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6634452b4c}} (increase uWSGI buffer)

=== 2020-04-18 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c815a210bd}} (Hebrew nouns)
* 17:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|33c3ac264e}} (fix english-adverb edit mode)
* 11:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2959ebf637}} (fix duplicates in advanced mode)

=== 2020-04-14 ===
* 20:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|44b5df2897}} (edit mode: show lemma, show conflicts, add missing statements)

=== 2020-04-13 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fe2118d4e}} (python3.7)
* 22:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ab7f751ba6}} (edit mode)

=== 2020-02-26 ===
* 00:22 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2020-01-28 ===
* 00:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61fe7e59fb}} (typofix)
* 00:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e0e916e0a5}} (more Persian translations and RTL fixes)

=== 2020-01-27 ===
* 23:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|54b9e37118}} (more RTL fixes)
* 23:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72ec256823}} (Persian nouns and verbs) [actually happened ~30mins ago, forgot to log]

=== 2020-01-15 ===
* 00:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bc1d49c202}} (better CSRF error handling, [[phab:T242573|T242573]])

=== 2020-01-14 ===
* 00:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|242c25810b}} (clarify Spanish verbs)

=== 2020-01-12 ===
* 14:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|edcbc10ae9}} (Spanish verbs)

=== 2020-01-11 ===
* 17:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d9619cb473}} (Danish nouns and verbs)
* 14:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4a20b4b95e}} (Czech perfective verbs)
* 14:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8da9227b52}} (fix typos in Czech adjective template)

=== 2019-11-30 ===
* 13:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2f5a8ccc2e}} (update english-verb)

=== 2019-11-21 ===
* 22:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|13cf2696b9}} (reorder)
* 22:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|89ad1e816c}} (Basque verbs)

=== 2019-11-11 ===
* 23:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cd4239904a}} (work around [[phab:T230833|T230833]])
* 21:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8b53b417c1}} (fixes to Kurdish (Kurmancî))
* 17:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fe31bd9aa6}} (message syntax fix)

=== 2019-11-10 ===
* 19:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9d736fe2f6}} (Kurdish Kurmancî nouns)
* 15:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|29e549fe31}} (Malayalam nouns)

=== 2019-10-27 ===
* 22:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fc68fabb5}} (lexeme IDs in bulk mode)

=== 2019-10-16 ===
* 22:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b480b6d07e}} (Czech translations + adjectives with more forms)

=== 2019-10-07 ===
* 22:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ce8ba2b234}} (add plural grammatical feature to Ukrainian plurale tantum forms)

=== 2019-09-30 ===
* 22:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19bf4e3347}} (remove PHP_ENGINE cookie)

=== 2019-08-28 ===
* 23:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a053e9a36e}} (update Swedish translations)

=== 2019-08-22 ===
* 22:53 wm-bot: &lt;lucaswerkmeister&gt; deployed 60cf696645v (minor bulk mode improvements)
* 22:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f4fd72ab72}} (bulk mode improvements)

=== 2019-08-20 ===
* 20:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|938075faf2}} (bulk mode)

=== 2019-08-11 ===
* 11:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|09a3ac6b64}} (Swedish absolute adjectives)

=== 2019-08-02 ===
* 21:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a4d699fbcb}} (fix item ID after merge)

=== 2019-07-24 ===
* 12:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f0883f1ebc}} (templates API)

=== 2019-07-07 ===
* 18:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|50a70b3590}} (Swedish verbs)
* 13:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9a148c8cc5}} (add statements when editing existing lexeme)
* 12:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a8242673b9}} (use jsonify)
* 12:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|994b980655}} (CORS for duplicates API)

=== 2019-07-06 ===
* 22:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b0f39bb09b}} (API to match lexemes to templates)

=== 2019-06-26 ===
* 20:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e74ff290cc}} (duplicates API bug fix) [actually deployed 2 hours ago, forgot to log]

=== 2019-06-24 ===
* 22:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e937ff5839}} (autocapitalize=&quot;off&quot; on form)
* 22:44 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted experimental change (autocapitalize=&quot;off&quot; on form and inputs)
* 22:29 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted experimental change (autocapitalize=&quot;off&quot; on form rather than inputs)
* 22:14 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted experimental change (autocapitalize=&quot;off&quot; on inputs)
* 21:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|07b05a6858}} (Portuguese verbs)

=== 2019-06-14 ===
* 19:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c48127f696}} (update Russian translations)
* 00:38 wm-bot: &lt;lucaswerkmeister&gt; kubectl delete deployment lexeme-forms.purge-all-lexemes # [[phab:T225510|T225510]] done

=== 2019-06-12 ===
* 08:48 wm-bot: &lt;lucaswerkmeister&gt; kubectl create -f deployment-purge-all-lexemes.yaml # [[phab:T225510|T225510]]

=== 2019-06-10 ===
* 19:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|645886b3a8}} (update German translations)
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|846100f8d9}} (update Czech translations)
* 12:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fe6cc3a79b}} (improved forms/senses message for duplicates)

=== 2019-06-09 ===
* 23:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c88de6348}} (number of forms/senses for duplicates)

=== 2019-06-08 ===
* 14:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f09dfd20a1}} (Dutch nouns)
* 14:00 wm-bot: &lt;lucaswerkmeister&gt; git remote add github https://github.com/lucaswerkmeister/tool-lexeme-forms.git # work around [[phab:T224677|T224677]]
* 12:17 wm-bot: &lt;lucaswerkmeister&gt; restarted webservice after redirect loop

=== 2019-05-20 ===
* 09:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|496a928b67}} (switch to Python 3.5), including venv rebuild
* 08:52 wm-bot: &lt;lucaswerkmeister&gt; stopping webserver for Python 3.5 upgrade


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>t97e9jap0spv4y5xxntl0b0olwffzdb</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.wd-image-positions/SAL</title>
    <ns>498</ns>
    <id>443947</id>
    <revision>
      <id>2320871</id>
      <parentid>2311600</parentid>
      <timestamp>2025-07-07T06:31:20Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~lucaswerkmeister@tools-bastion-13: deployed d1f4beb35e (l10n updates: af, ru, tr)</comment>
      <origin>2320871</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="20959" sha1="d6d6bv5ijn13fnw4hxp1lhe0iixjk9t" xml:space="preserve">=== 2025-07-07 ===
* 06:31 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d1f4beb35e}} (l10n updates: af, ru, tr)

=== 2025-06-11 ===
* 22:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9d8920826c}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]])

=== 2025-05-26 ===
* 16:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9cea85d255}} (l10n updates: tr)

=== 2025-05-15 ===
* 21:51 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|17a80202ae}} (l10n updates: it)

=== 2025-05-08 ===
* 13:07 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a330358f39}} (l10n updates: hu, ka); also includes {{Gerrit|8865bb0c67}} (l10n updates: kaa) – apparently I forgot to `git rebase` after `git fetch` last time 🤦

=== 2025-04-24 ===
* 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8865bb0c67}} (l10n updates: kaa)

=== 2025-04-21 ===
* 18:14 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|840b383c90}} (upgrade dependencies, including Flask 3.1.0 and toolforge-i18n 0.1.2)
* 16:46 wmbot~lucaswerkmeister@tools-bastion-13: webservice restart # clear out a stuck broken variable in toolforge-i18n

=== 2025-04-17 ===
* 19:04 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|919afec938}} (l10n updates: es)

=== 2025-04-14 ===
* 18:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0b184b8268}} (l10n updates: af)

=== 2025-03-07 ===
* 19:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b8022cddca}} (l10n updates: tr)

=== 2025-02-27 ===
* 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f053d010c6}} (l10n updates: sah)

=== 2025-02-13 ===
* 21:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2f4a78c0ad}} (l10n updates: rki)

=== 2025-01-30 ===
* 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|223cf553fb}} (l10n updates: ar)

=== 2025-01-09 ===
* 12:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6e55549156}} (l10n updates: lb)

=== 2025-01-06 ===
* 20:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b7c6f2e7e5}} (l10n updates: cs, sv)

=== 2024-12-16 ===
* 20:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5bea039bac}} (l10n updates: bbc-latn, lb)

=== 2024-11-28 ===
* 21:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|602dbded5b}} (l10n updates: tcy)

=== 2024-11-21 ===
* 22:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3486d60faf}} (l10n updates: tcy)

=== 2024-11-18 ===
* 19:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bfcbf544da}} (remove canonical from service.template)
* 19:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1527c5d7c8}} (l10n updates: krc)

=== 2024-11-11 ===
* 19:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a6d0e2ad43}} (l10n updates: ar)

=== 2024-11-05 ===
* 18:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|33511f4de7}} (l10n updates: it, lb, sr-ec, zh-hans, zh-hant; had been blocked by [[phab:T373807|T373807]])

=== 2024-10-25 ===
* 19:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4bf78a4c42}} (upgrade dependencies, including Werkzeug 3.0.6)

=== 2024-10-13 ===
* 11:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a4d0774ea2}} (upgrade dependencies, including MarkupSafe 3.0); also deploys {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) which I previously accidentally “deployed” and logged in quickcategories instead, oops :D

=== 2024-10-03 ===
* 16:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|694704b8ab}} (upgrade dependencies, including toolforge_i18n and Werkzeug 3.0.4)

=== 2024-09-01 ===
* 14:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6430a15668}} (l10n updates: qqq fix)

=== 2024-08-05 ===
* 13:35 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|898795588f}} (upgrade toolforge_i18n to 0.0.7)

=== 2024-08-01 ===
* 18:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|95846f654a}} (l10n updates: zh-hans)

=== 2024-07-31 ===
* 19:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5540ef17c9}} (upgrade toolforge_i18n to 0.0.6; also upgrade pip to 24.2)

=== 2024-07-30 ===
* 17:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3f5ba1a732}} (l10n updates: wal)

=== 2024-07-25 ===
* 19:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f515834cff}} (l10n updates: de)

=== 2024-07-22 ===
* 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|54ad2210a2}} (l10n updates: ar, kaa)

=== 2024-07-21 ===
* 18:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a291d8f4ab}} (upgrade toolforge_i18n to 0.0.5; also upgrade pip to 24.1.2)

=== 2024-07-19 ===
* 17:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4757a1f91e}} (l100n updates: de, nb, uk)

=== 2024-07-08 ===
* 18:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ee1bf2fdeb}} (l10n updates: ms, sk)

=== 2024-07-07 ===
* 18:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|01d9085f1a}} (upgrade toolforge_i18n to 0.0.2; also upgrade pip from 24.0 to 24.1.1)

=== 2024-07-01 ===
* 17:09 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|64c8192524}} (l10n updates: nb)

=== 2024-06-17 ===
* 16:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1b8e7b6b05}} (l10n updates: it, nl)

=== 2024-06-15 ===
* 12:36 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|887589b017}} (install toolforge_i18n from PyPI)

=== 2024-06-14 ===
* 17:51 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|62f5575dbd}} (fix logout, noticed while testing [[phab:T367188|T367188]])
* 17:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|606e0ca438}} ([[phab:T367188|T367188]])

=== 2024-06-10 ===
* 18:37 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|dcf0c5040e}} (l10n updates: ru)

=== 2024-06-07 ===
* 09:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|88402067dd}} (update GitHub actions), pulled without webservice restart
* 09:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|24a82d8701}} (search items in interface language)

=== 2024-06-06 ===
* 20:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ea1e8c10e2}} (fix language fallback in edit interface)
* 20:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|72a68851e0}} (fix TIFF)
* 19:55 wmbot~lucaswerkmeister@tools-bastion-13: webservice restart # was unresponsive, idk why

=== 2024-06-03 ===
* 12:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|aa319bc391}} (l10n updates: fr, he, qqq)

=== 2024-05-30 ===
* 17:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|53074960ca}} (l10n updates: es, fr)

=== 2024-05-27 ===
* 17:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|40f6405ab3}} (l10n updates: fr, ko, ru)

=== 2024-05-23 ===
* 20:10 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b82dc1062f}} (l10n updates: de, gl, ko, lb, mk, sl, sr-ec)

=== 2024-05-20 ===
* 13:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2965e37ca2}} (l10n updates: fr, ia, it, ms, sr-ec, zh-hans, zh-hant)

=== 2024-05-19 ===
* 16:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0486d8ade3}} (make rest of tool translatable; resolves [[phab:T363626|T363626]])
* 14:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|903c13a18a}} (make editing interface translatable, [[phab:T363626|T363626]])
* 13:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bdd23c3081}} (fix template)
* 13:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bd8f091ad1}} (make “… with no region specified” translatable, [[phab:T363626|T363626]])

=== 2024-05-18 ===
* 14:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|108e8a5fa6}} (fix csrf_token check)

=== 2024-05-16 ===
* 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f374ee202}} (l10n updates: es, zh-hans)

=== 2024-05-13 ===
* 18:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1ca9f6af8c}} (l10n updates: gl, he, ia, lb, mk, nl, sl, sr-ec, zh-hant)

=== 2024-05-09 ===
* 15:11 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e8d47d1cb7}} (improve toolforge_i18n and upgrade dependencies for newer Babel)

=== 2024-05-06 ===
* 20:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e5e9116fc9}} (optimize GitLab CI; pulled without webservice restart)
* 18:27 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e8d438134b}} (add GitLab CI for [[phab:T363626|T363626]]; pulled without webservice restart)
* 15:45 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8b677d25a9}} (README update, pulled without webservice restart)
* 15:12 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e4205d9cee}} (l10n updates: de, es, fr, it, zh-hant)

=== 2024-05-05 ===
* 10:47 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|6b0e493e20}} (fix language setting, [[phab:T363626|T363626]])
* 10:42 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|73baa98186}} (l10n updates: de, eu, fi, ko, mk, nb, nl; [[phab:T363626|T363626]])
* 09:53 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c6407a31ae}} (make “logged in as” translatable, [[phab:T363626|T363626]])
* 09:48 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|17817c0cad}} (GitHub actions fix, pulled without webservice restart, [[phab:T363626|T363626]])
* 09:41 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|279399cf63}} (fix double “image scale” text from language fallback)
* 09:35 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|4da2a2bc42}} (settings page, [[phab:T363626|T363626]])

=== 2024-05-04 ===
* 12:24 wmbot~lucaswerkmeister@tools-sgebastion-10: added l10n-bot as developer member on GitLab (ca. 30 minutes ago, but logging now for the record) ([[phab:T363626|T363626]])
* 12:24 wmbot~lucaswerkmeister@tools-sgebastion-10: added l10n-bot as developer member on GitLab (ca. 30 minutes ago, but logging now for the record)
* 12:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|418ca66477}} (make translatable using toolforge_i18n: [[phab:T363626|T363626]])

=== 2024-05-03 ===
* 17:15 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|3204d71b37}} (upgrade dependencies for Python 3.12 compat; also upgraded pip&lt;nowiki&gt;{&lt;/nowiki&gt;,-tools&lt;nowiki&gt;}&lt;/nowiki&gt; and wheel while I’m at it)

=== 2024-04-08 ===
* 18:06 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|f6dfc23eec}} (make session permanent after login) + {{Gerrit|7776444690}} (use property labels on index page)

=== 2023-12-11 ===
* 01:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bf2f609917}} (fix Vue template)

=== 2023-10-25 ===
* 20:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7aec0c0b10}} (use Codex 1.0.0)
* 18:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f7a426b0de}} (Werkzeug 3.0.1)

=== 2023-10-03 ===
* 10:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9d55c80b99}} (update dependencies, Flask+Werkzeug 3) [actually ~10 mins ago but I forgot to log it]

=== 2023-07-23 ===
* 15:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|143b86e1b3}} (disable autoCrop)
* 15:35 wm-bot: &lt;lucaswerkmeister&gt; *actually* deployed {{Gerrit|a7c924557b}} (forgot to git rebase)
* 13:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7c924557b}} (fix color of newly added regions)
* 13:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a479974a9d}} (only load edit interface when logged in)

=== 2023-07-19 ===
* 18:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0e0deb1a49}} (reimplement “add new depicted” form in Codex)

=== 2023-07-15 ===
* 10:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|81999c31af}} (Python 3.11)

=== 2023-05-01 ===
* 17:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3cfa978e37}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq)

=== 2023-04-26 ===
* 19:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c6cf151896}} (update dependencies, mainly Flask+Werkzeug 2.3)

=== 2023-02-14 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d498c2ecbd}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes)

=== 2022-09-10 ===
* 18:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8df38d8f43}} (README fix, pulled without webservice restart)
* 16:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ec75eef080}} (gitlab lucaswerkmeister/ → gitlab toolforge-repos/)

=== 2022-07-24 ===
* 16:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|07781279a4}} (fix images with question marks in the title, in a third place)
* 16:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4a08744dc4}} (fix images with question marks in the title, in another place)
* 15:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4793d848d8}} (fix images with question marks in the title)

=== 2022-05-26 ===
* 12:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e878def771}} (two CSS improvements)
* 11:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|43f0101e20}} (Bootstrap 5.1)

=== 2022-05-22 ===
* 20:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a846c4f7f4}} (refactor depicted.css)
* 13:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|15ade38230}} (Diffusion → GitLab)
* 12:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2264a649ae}} (flake8, updated dependencies)
* 09:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4630899d1e}} (tweak message)
* 09:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2bb7a08f06}} (abort on invalid region instead of crashing)

=== 2022-05-20 ===
* 17:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eeb434ac0b}} (add cancel buttons)
* 15:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7abe262b2a}} (remove unused WIP code)
* 13:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f039f87ae0}} (more supported input formats on index page, especially URLs)
* 10:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|010a68cafe}} (image height &lt; viewport height)
* 09:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fc87dbcf3}} (scroll only image when scaling)

=== 2022-05-17 ===
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|803f7f1f3a}} (info when users can’t edit due to noscript or missing login)

=== 2022-05-15 ===
* 16:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b2fea9010f}} (image scaling support, better width+height+srcset)

=== 2022-05-01 ===
* 17:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ecae67b924}} (fix item ID input handling)
* 15:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|36f85185cb}} (remove QuickStatements support, add strict mode)
* 14:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3fc93364c5}} (add named place on map statements; update cropper.js)

=== 2022-04-30 ===
* 18:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e48c179d47}} (initial support for named place on map, can show and edit regions but not yet add statements)
* 13:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a37bcf7875}} (proper User-Agent)
* 12:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|854eb65b5f}} (use pip-tools; updated some packages, including Flask 2.0.1 → 2.1.2)

=== 2021-11-24 ===
* 13:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61e649c853}} (fix error for some IIIF manifests)

=== 2021-11-03 ===
* 20:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|620e07e107}} (refactoring, no functional change hopefully)

=== 2021-10-28 ===
* 23:21 wm-bot: &lt;lucaswerkmeister&gt; tool is back up but the pre-2021 logs are gone instead of backed up because the container doesn’t have zstd and I didn’t notice in time :(
* 23:19 wm-bot: &lt;lucaswerkmeister&gt; briefly stopping tool (few minutes) to cycle the uwsgi.log

=== 2021-09-25 ===
* 14:46 wm-bot: &lt;lucaswerkmeister&gt; removed old venv-3.5 venv-3.7
* 14:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8797b176c6}} (Python 3.9)

=== 2021-08-02 ===
* 20:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61358b4346}} (one more title and a crash fix)
* 19:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57b861aaf8}} (useful page title)

=== 2021-07-18 ===
* 19:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eb1f1e04bf}} (update config loading; also updated the venv, including Flask v2)

=== 2021-03-01 ===
* 21:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bf35152db8}} (GitHub actions; adds pytest to venv)

=== 2021-02-28 ===
* 18:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ae6a228597}} (better OAuth error handling)

=== 2021-02-19 ===
* 20:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|73a61f6709}} (avoid mwoauth.identify)

=== 2021-02-16 ===
* 20:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|83700bdd07}} (add skip link)
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e791f3d956}} (Bootstrap update)

=== 2020-10-17 ===
* 14:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c163184b3a}} (more durable CSRF tokens)

=== 2020-08-08 ===
* 16:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6ec61155ed}} (JS refactoring)

=== 2020-08-07 ===
* 20:22 wm-bot: &lt;lucaswerkmeister&gt; deployed  {{Gerrit|e7fd7f5c82}} (support TIFF etc.)

=== 2020-07-25 ===
* 18:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0dea27dbbd}} (deprecate QuickStatements, minor fixes)

=== 2020-07-22 ===
* 22:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9eb2aa216d}} (no edit region without regions)
* 22:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aa97ea1589}} (Esc for editing regions)
* 21:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6b34c5bb7b}} (editing regions)

=== 2020-06-15 ===
* 21:05 wm-bot: &lt;lucaswerkmeister&gt; renamed default branch from master to main

=== 2020-05-30 ===
* 14:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ff77d74e3f}} (add somevalue depicts statements)

=== 2020-04-25 ===
* 17:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03e5c7016a}} (fix image.js)
* 17:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5fca9d7d94}} (toolforge.org)
* 16:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|225aea85f1}} (python3.7)

=== 2020-03-28 ===
* 20:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|99ed67d128}} (fix rotated JPEGs some more)
* 14:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|152af31cca}} (fix rotated JPEGs)

=== 2020-03-21 ===
* 14:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|52678de195}} (handle missing/invalid titles and langcode improvement)

=== 2020-02-28 ===
* 21:03 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2020-02-09 ===
* 11:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|97900da63e}} (another language code improvement)
* 06:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5aa0cf4ccb}} (better language code handling)

=== 2019-12-01 ===
* 19:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b72b69d7cb}} (item selector for adding new statements)

=== 2019-11-29 ===
* 23:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f07ce73792}} (fixes for somevalue/novalue and user script)

=== 2019-11-23 ===
* 21:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5eaaa61ffd}} (add new depicteds, some other improvements)
* 14:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|14459bcd4e}} (edit somevalue/novalue)
* 14:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c0b3450603}} (remove legacy API)
* 14:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|448bf42367}} (show depicted somevalue/novalue, and committed no-longer-temporary experiment for OAuth errors)

=== 2019-11-17 ===
* 18:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ef3af6b29f}} (clean up underscore handling; temporary experiment for OAuth errors still in effect)

=== 2019-11-15 ===
* 20:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57735fe450}} (fix overflow behavior; temporary experiment for OAuth errors still in effect)

=== 2019-11-01 ===
* 11:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|520c618b5a}} (work around [[phab:T222159|T222159]]); temporary experiment still in effect

=== 2019-10-25 ===
* 21:07 wm-bot: &lt;lucaswerkmeister&gt; deployed temporary experiment (clear session on OAuth callback error)
* 15:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7a4982705c}} (improve headings)
* 14:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2d28f55912}} (optimize cropper image loading)

=== 2019-10-24 ===
* 20:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b5dd0fdb31}} (bugfix)

=== 2019-10-23 ===
* 21:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ceb0b0cf8f}} (Structured Data on Commons support, use cropperjs; also tried a general pip upgrade but that broke OAuth login so downgraded to previous versions, except pip itself)

=== 2019-05-20 ===
* 09:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dfaa0c4093}} (switch to Python 3.5), including venv rebuild
* 09:23 wm-bot: &lt;lucaswerkmeister&gt; stopping webserver for Python 3.5 upgrade


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>d6d6bv5ijn13fnw4hxp1lhe0iixjk9t</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.ranker/SAL</title>
    <ns>498</ns>
    <id>447034</id>
    <revision>
      <id>2320869</id>
      <parentid>2311478</parentid>
      <timestamp>2025-07-07T06:29:18Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~lucaswerkmeister@tools-bastion-13: deployed e059817c66 (l10n updates: fr, he, pt)</comment>
      <origin>2320869</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="11464" sha1="9zj6yti583vmxmm3qpmxbwj2elijz69" xml:space="preserve">=== 2025-07-07 ===
* 06:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e059817c66}} (l10n updates: fr, he, pt)

=== 2025-06-11 ===
* 17:41 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a167fc8e71}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]])

=== 2025-05-26 ===
* 16:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5e4572a330}} (l10n updates: it)

=== 2025-05-08 ===
* 13:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d1e60efda3}} (l10n updates: nl)

=== 2025-04-24 ===
* 19:21 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9be26a5e6}} (l10n updates: ru)

=== 2025-04-21 ===
* 18:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b0a9db1622}} (upgrade dependencies, including Flask 3.1.0 and toolforge-i18n 0.1.2)

=== 2025-04-18 ===
* 14:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|37e5987b2b}} (l10n updates: fr)

=== 2025-04-17 ===
* 19:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|44c3d7820d}} (l10n updates: zh-hans)

=== 2025-04-10 ===
* 12:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0f074e3dd6}} (l10n updates: es)

=== 2025-04-07 ===
* 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|348dc8edc7}} (l10n updates: es, zh-hant)

=== 2025-04-04 ===
* 19:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|79c8ebeac5}} (Bootstrap 5.3 and better RTL support)
* 11:36 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ee6adc189c}} (l10n updates: ar)

=== 2025-04-01 ===
* 12:04 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|32a85a522b}} (l10n updates: nl)

=== 2025-03-28 ===
* 13:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d95b93c7b8}} (l10n updates: mk)

=== 2025-03-20 ===
* 20:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b1be7f07fa}} (l10n updates: ru)

=== 2025-03-13 ===
* 22:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5098bca3bb}} (l10n updates: el)

=== 2025-03-07 ===
* 19:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5ced6327b8}} (l10n updates: nl)

=== 2025-02-24 ===
* 19:09 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80fa92877e}} (l10n updates: diq)

=== 2025-02-17 ===
* 18:31 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|43647bd6bb}} (l10n updates: sr-ec)

=== 2025-02-13 ===
* 21:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7630a9250a}} (l10n updates: ko, lb)

=== 2025-02-12 ===
* 18:57 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|778c5bbd38}} (add settings page for [[phab:T384061|T384061]])

=== 2025-02-11 ===
* 20:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|49f1c8b721}} (l10n updaets: qqq)

=== 2025-02-10 ===
* 19:04 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|94e4c884d3}} (l10n updates: diq)
* 19:04 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|94e4c884d3}} (l10n updates: diq)

=== 2025-02-07 ===
* 09:45 wmbot~lucaswerkmeister@tools-bastion-13: (new code version is now live and has been for ~8h thanks to [[phab:T385847|T385847]] being fixed)
* 00:26 wmbot~lucaswerkmeister@tools-bastion-13: (new code / l10n version is not actually live yet due to [[phab:T385847|T385847]])
* 00:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1abc7122fa}} (l10n updates: lb, skr-arab)

=== 2025-01-31 ===
* 13:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d65fa5888b}} (fix language fallback)
* 13:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|19c821857d}} (l10n updates: fi, ko, lb, skr-arab, sr-ec; [[phab:T384061|T384061]])

=== 2025-01-21 ===
* 10:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c326b10184}} (make tool translatable: [[phab:T384061|T384061]]); also update pip and wheel

=== 2024-10-25 ===
* 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c034355650}} (upgrade dependencies, including Werkzeug 3.0.6)

=== 2024-09-01 ===
* 12:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3c42814e32}} (upgrade dependencies; also upgraded pip + wheel + pip-tools in the venv)

=== 2024-04-08 ===
* 18:20 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|b67a349bee}} (make session permanent after login)

=== 2023-12-03 ===
* 12:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|da71d5402e}} (force English for wbformatvalue+wbformatentities, [[phab:T345881|T345881]]

=== 2023-10-25 ===
* 18:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3bd9718d9a}} (Werkzeug 3.0.1)

=== 2023-10-03 ===
* 14:32 wm-bot: &lt;lucaswerkmeister&gt; rm -rf www/python/venv-3.9/ # unused
* 14:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|358730fbbf}} (update dependencies, Flask+Werkzeug 3)

=== 2023-07-15 ===
* 15:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0e3bf213e8}} (cleanup typings and update github actions)
* 15:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1de5a9e1f1}} (Python 3.11)

=== 2023-05-01 ===
* 23:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5e36e13fb0}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq)

=== 2023-04-29 ===
* 18:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b9dfdfe2c4}} (upgrade dependencies, Flask/Werkzeug 2.3)

=== 2023-03-13 ===
* 22:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|11379c37a5}} (improve error handling for invalid statement IDs)
* 21:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b3e9a4cb0a}} (fix form/sense statement IDs)

=== 2023-02-18 ===
* 17:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|025d1d79dd}} (skip no-op edits without API request)

=== 2023-02-14 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a78bc0cfd0}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes)

=== 2022-09-25 ===
* 19:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aba490b308}} (support reason for preferred / deprecated rank)
* 11:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aac8deeda2}} (add code documentation, no-op)
* 10:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4996e649fb}} (clarify code documentation, no-op)

=== 2022-09-24 ===
* 20:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|26a3edad41}} (highlight reason for preferred / deprecated rank)

=== 2022-09-11 ===
* 14:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7802bd01c3}} (format values as HTML)
* 14:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|884f873fd0}} (manage requirements.txt using pip-tools)

=== 2022-09-10 ===
* 18:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0578fa6519}} (diffusion → gitlab)

=== 2022-02-21 ===
* 18:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|67f40e12b8}} (tweak error message)
* 18:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|561450bdbd}} (remove Commons query support because WCQS beta 2 requires authentication)

=== 2021-11-14 ===
* 18:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|98be42d44e}} (only send edited statements to API, saves traffic and avoids errors due to unrelated statements)

=== 2021-10-13 ===
* 23:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3fd9ab9f20}} (remove type ignore comments), updated dependencies including Flask 2.0.2, fully restarted webservice (stop/start) to avoid label issues

=== 2021-10-06 ===
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3540e2a083}} (🌈 navbar)

=== 2021-09-25 ===
* 14:48 wm-bot: &lt;lucaswerkmeister&gt; removed old venv-3.7

=== 2021-08-15 ===
* 18:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9235b38189}} (Python 3.9, CC [[phab:T284590|T284590]])

=== 2021-07-20 ===
* 19:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f309990d5b}} (update config loading code; also upgraded venv, e.g. Flask v2)

=== 2021-06-06 ===
* 10:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1d9f0d09c8}} (mypy fix)

=== 2021-05-23 ===
* 10:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ab07154319}} (restore batch links on index, working around Chromium layout issue)
* 10:04 wm-bot: &lt;lucaswerkmeister&gt; rolled back to {{Gerrit|dc734361cf}} (layout issues in Chromium, investigating)
* 09:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ffa1c2a5a8}} (batch mode on index page)
* 09:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dc734361cf}} (only mypy comments but restarted the webservice anyways just in case)

=== 2021-05-22 ===
* 22:04 wm-bot: &lt;lucaswerkmeister&gt; pulled {{Gerrit|efa1cecac0}} (README update, no webservice restart)
* 10:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|38b8f68887}} (cross-links between batch modes)

=== 2021-05-20 ===
* 21:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a6c373eb70}} (query individual batch mode)
* 17:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8be4c55231}} (query collective batch mode)
* 17:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fc9e2977c6}} (typing fixes)

=== 2021-05-17 ===
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95f2125dc4}} (edit summary fixes)

=== 2021-05-16 ===
* 18:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8784dddb07}} (batch mode, rank per individual statement)
* 13:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72ec33e6f2}} (minor improvements)

=== 2021-05-15 ===
* 18:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|93d904cb7e}} (batch mode, list+collective version)

=== 2021-05-10 ===
* 17:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c78f9f4b5}} (remove dead code)

=== 2021-02-28 ===
* 19:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bbca6e5b8e}} (better OAuth error handling)

=== 2021-02-19 ===
* 20:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a98e831449}} (avoid mwoauth.identify)

=== 2021-02-16 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1ff5cbea92}} (add skip link)
* 19:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|140d8f00fd}} (Bootstrap update)

=== 2021-02-10 ===
* 20:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6a164dddea}} (&lt;script defer&gt;)

=== 2021-01-31 ===
* 15:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|981843b704}} (work around [[phab:T222159|T222159]])
* 15:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ef54a0b1a8}} (handle missing entity error)
* 14:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a6392a7c30}} (File:… as input on index page)

=== 2021-01-30 ===
* 13:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|50f5f35cb4}} (singular/plural)
* 12:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|64cc29192a}} (format entity IDs)

=== 2021-01-27 ===
* 20:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c434d9994c}} (remember current page on login)
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fd1f5959d}} (move login banner)

=== 2021-01-26 ===
* 20:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1e9fbd00dd}} (regex fix)

=== 2021-01-24 ===
* 18:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61293fdc50}} (code style only)
* 18:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|996b9471ec}} (back button for no statements)
* 18:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7c6f523206}} (custom summary for increment)
* 17:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e6eed96fff}} (custom edit summary and other improvements)

=== 2021-01-16 ===
* 19:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|20cf18c1bc}} (code cleanups)
* 17:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8e2c9a4b34}} (cleanup)
* 17:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|97774ca30c}} (initial deployment) about ten minutes ago


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>9zj6yti583vmxmm3qpmxbwj2elijz69</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2320867</id>
      <parentid>2320859</parentid>
      <timestamp>2025-07-07T00:01:38Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2320867</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="544" sha1="03nyz2vlalnladsv5r7hmy0b5fiw987" xml:space="preserve">{{/Header}}
== Today (2025-07-07) ==
== Yesterday (2025-07-06) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| pc3 || [[phab:T378715|Possibility to transition some codfw data persistence hosts to 10G (T378715)]] (ladsgroup)
|-
| pc4 || [[phab:T378715|Possibility to transition some codfw data persistence hosts to 10G (T378715)]] (ladsgroup)
|-
| s5 || 
* [[phab:T395241|Login (T395241)]] (fceratto)
* [[phab:T398594|Switchover s5 master (db2213 -&amp;gt; db2192) (T398594)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>03nyz2vlalnladsv5r7hmy0b5fiw987</sha1>
    </revision>
    <revision>
      <id>2320879</id>
      <parentid>2320867</parentid>
      <timestamp>2025-07-07T07:23:29Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2320879</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="846" sha1="pimy2yvbexrza9inxzpdyr6ton9417n" xml:space="preserve">{{/Header}}
== Today (2025-07-07) ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| x1 || [[phab:T397612|Switchover x1 master (db1237 -&amp;gt; db1220) (T397612)]] (marostegui)
|-
|}
== Yesterday (2025-07-06) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| x1 || [[phab:T397612|Switchover x1 master (db1237 -&amp;gt; db1220) (T397612)]] (marostegui)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| pc3 || [[phab:T378715|Possibility to transition some codfw data persistence hosts to 10G (T378715)]] (ladsgroup)
|-
| pc4 || [[phab:T378715|Possibility to transition some codfw data persistence hosts to 10G (T378715)]] (ladsgroup)
|-
| s5 || 
* [[phab:T395241|Login (T395241)]] (fceratto)
* [[phab:T398594|Switchover s5 master (db2213 -&amp;gt; db2192) (T398594)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>pimy2yvbexrza9inxzpdyr6ton9417n</sha1>
    </revision>
    <revision>
      <id>2320935</id>
      <parentid>2320879</parentid>
      <timestamp>2025-07-07T11:57:04Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2320935</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1169" sha1="hhlmvatobqpkkbm7w8ysklzzwsgxw6m" xml:space="preserve">{{/Header}}
== Today (2025-07-07) ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| x1 || [[phab:T397612|Switchover x1 master (db1237 -&amp;gt; db1220) (T397612)]] (marostegui)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| s1 || [[phab:T398433|lsw1-a8-codfw: fpc0 PFE Statistics received unknown trigger (type Semaphore, id 0) (T398433)]] (ladsgroup)
|-
|}
== Yesterday (2025-07-06) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| x1 || [[phab:T397612|Switchover x1 master (db1237 -&amp;gt; db1220) (T397612)]] (marostegui)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| pc3 || [[phab:T378715|Possibility to transition some codfw data persistence hosts to 10G (T378715)]] (ladsgroup)
|-
| pc4 || [[phab:T378715|Possibility to transition some codfw data persistence hosts to 10G (T378715)]] (ladsgroup)
|-
| s1 || [[phab:T398433|lsw1-a8-codfw: fpc0 PFE Statistics received unknown trigger (type Semaphore, id 0) (T398433)]] (ladsgroup)
|-
| s5 || 
* [[phab:T395241|Login (T395241)]] (fceratto)
* [[phab:T398594|Switchover s5 master (db2213 -&amp;gt; db2192) (T398594)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>hhlmvatobqpkkbm7w8ysklzzwsgxw6m</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:Renamed user c94112c361f80e601af4d9516826efa3</title>
    <ns>3</ns>
    <id>454542</id>
    <revision>
      <id>2320863</id>
      <parentid>2271090</parentid>
      <timestamp>2025-07-06T13:50:03Z</timestamp>
      <contributor>
        <username>XXBlackburnXx</username>
        <id>38000</id>
      </contributor>
      <minor />
      <comment>XXBlackburnXx moved page [[User talk:NNNH]] to [[User talk:Renamed user c94112c361f80e601af4d9516826efa3]] without leaving a redirect: Automatically moved page while renaming the user &quot;[[Special:CentralAuth/NNNH|NNNH]]&quot; to &quot;[[Special:CentralAuth/Renamed user c94112c361f80e601af4d9516826efa3|Renamed user c94112c361f80e601af4d9516826efa3]]&quot;</comment>
      <origin>2271090</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1190" sha1="0wjrh0br4u4s5v9psz2p1thuh1vey1t" xml:space="preserve">== Welcome to Toolforge! ==
Hello Winzekter986, welcome to the Toolforge project!  Your request for access was processed, and you should be able to use ssh to connect to &lt;tt&gt;login.toolforge.org&lt;/tt&gt;. You will need to logout and login again at https://toolsadmin.wikimedia.org/ to activate your new permissions there.

Check the [[Help:Toolforge|Toolforge help page]] for tips on using your account.  You can also ask questions in our IRC channel at {{irc|wikimedia-cloud}} or send an e-mail to our mailing list &lt;tt&gt;cloud@lists.wikimedia.org&lt;/tt&gt;.

Thank you, and have fun making Tools! --[[User:StrikerBot|StrikerBot]] ([[User talk:StrikerBot|talk]]) 08:05, 27 May 2024 (UTC)
== Wikitech account renamed and attached to SUL ==

Your Wikitech account has been renamed to match the SUL account you associated it with using toolsadmin.wikimedia.org or idm.wikimedia.org. Following the rename your Wikitech account was attached to your SUL account. You should now be able to login to wikitech.wikimedia.org using your SUL account in the same way you would login to any other Wikimedia project wiki. -- [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]]) 00:58, 11 February 2025 (UTC)</text>
      <sha1>0wjrh0br4u4s5v9psz2p1thuh1vey1t</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:J0rd1</title>
    <ns>3</ns>
    <id>457329</id>
    <revision>
      <id>2320866</id>
      <parentid>2270037</parentid>
      <timestamp>2025-07-06T20:34:33Z</timestamp>
      <contributor>
        <username>J0rd1</username>
        <id>37490</id>
      </contributor>
      <comment>Blanked the page</comment>
      <origin>2320866</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="0" sha1="phoiac9h4m842xq45sp7s6u21eteeq1" />
      <sha1>phoiac9h4m842xq45sp7s6u21eteeq1</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:Karacehennem</title>
    <ns>3</ns>
    <id>459034</id>
    <revision>
      <id>2320920</id>
      <timestamp>2025-07-07T11:06:51Z</timestamp>
      <contributor>
        <username>StrikerBot</username>
        <id>8475</id>
      </contributor>
      <comment>Welcome to Toolforge!</comment>
      <origin>2320920</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="675" sha1="bbi0bsu6mn94gmtxu34m15ue7sxgdam" xml:space="preserve">== Welcome to Toolforge! ==
Hello Karacehennem, welcome to the Toolforge project!  Your request for access was processed, and you should be able to use ssh to connect to &lt;tt&gt;login.toolforge.org&lt;/tt&gt;. You will need to logout and login again at https://toolsadmin.wikimedia.org/ to activate your new permissions there.

Check the [[Help:Toolforge|Toolforge help page]] for tips on using your account.  You can also ask questions in our IRC channel at {{irc|wikimedia-cloud}} or send an e-mail to our mailing list &lt;tt&gt;cloud@lists.wikimedia.org&lt;/tt&gt;.

Thank you, and have fun making Tools! --[[User:StrikerBot|StrikerBot]] ([[User talk:StrikerBot|talk]]) 11:06, 7 July 2025 (UTC)</text>
      <sha1>bbi0bsu6mn94gmtxu34m15ue7sxgdam</sha1>
    </revision>
  </page>
</mediawiki>